2025-09-17 21:58:17,593 INFO StreamThr :1208566 [internal.py:wandb_internal():86] W&B internal server running at pid: 1208566, started at: 2025-09-17 21:58:17.591274 2025-09-17 21:58:17,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status 2025-09-17 21:58:17,632 INFO WriterThread:1208566 [datastore.py:open_for_write():85] open: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/run-jdwsvk36.wandb 2025-09-17 21:58:17,634 DEBUG SenderThread:1208566 [sender.py:send():382] send: header 2025-09-17 21:58:17,716 DEBUG SenderThread:1208566 [sender.py:send():382] send: run 2025-09-17 21:58:18,506 INFO SenderThread:1208566 [dir_watcher.py:__init__():211] watching files in: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files 2025-09-17 21:58:18,506 INFO SenderThread:1208566 [sender.py:_start_run_threads():1136] run started: jdwsvk36 with start time 1758146297.591427 2025-09-17 21:58:18,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: check_version 2025-09-17 21:58:18,545 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: check_version 2025-09-17 21:58:18,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: run_start 2025-09-17 21:58:18,722 DEBUG HandlerThread:1208566 [system_info.py:__init__():32] System info init 2025-09-17 21:58:18,722 DEBUG HandlerThread:1208566 [system_info.py:__init__():47] System info init done 2025-09-17 21:58:18,722 INFO HandlerThread:1208566 [system_monitor.py:start():194] Starting system monitor 2025-09-17 21:58:18,722 INFO SystemMonitor:1208566 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-09-17 21:58:18,723 INFO HandlerThread:1208566 [system_monitor.py:probe():214] Collecting system info 2025-09-17 21:58:18,723 INFO SystemMonitor:1208566 [interfaces.py:start():190] Started cpu monitoring 2025-09-17 21:58:18,724 INFO SystemMonitor:1208566 [interfaces.py:start():190] Started disk monitoring 2025-09-17 21:58:18,725 INFO SystemMonitor:1208566 [interfaces.py:start():190] Started gpu monitoring 2025-09-17 21:58:18,725 INFO SystemMonitor:1208566 [interfaces.py:start():190] Started memory monitoring 2025-09-17 21:58:18,726 INFO SystemMonitor:1208566 [interfaces.py:start():190] Started network monitoring 2025-09-17 21:58:18,770 DEBUG HandlerThread:1208566 [system_info.py:probe():196] Probing system 2025-09-17 21:58:18,778 DEBUG HandlerThread:1208566 [system_info.py:_probe_git():181] Probing git 2025-09-17 21:58:18,835 DEBUG HandlerThread:1208566 [system_info.py:_probe_git():189] Probing git done 2025-09-17 21:58:18,835 DEBUG HandlerThread:1208566 [system_info.py:probe():244] Probing system done 2025-09-17 21:58:18,835 DEBUG HandlerThread:1208566 [system_monitor.py:probe():223] {'os': 'Linux-5.14.21-150500.55.97_13.0.78-cray_shasta_c-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2025-09-17T21:58:18.770902', 'startedAt': '2025-09-17T21:58:17.540962', 'docker': None, 'cuda': None, 'args': ('--yaml_config=./config/sfnonet.yaml', '--config=multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2', '--run_num=v0.1.0-seed77', '--amp_mode=bf16', '--checkpointing_level=0', '--multistep_count=2', '--h_parallel_size=4', '--w_parallel_size=1', '--initialization_seed=77', '--load_checkpoint=legacy', '--batch_size=32'), 'state': 'running', 'program': '-m makani.train', 'codePathLocal': None, 'git': {'remote': 'git@github.com:ankurmahesh/modulus-makani-fork.git', 'commit': '4b7dcddea9084b60c41957440a9f3e14f7d2567d'}, 'email': 'mahesh.ankur10@gmail.com', 'root': '/global/u2/a/amahesh/ms_finetune/modulus-makani-fork', 'host': 'nid008216', 'username': 'amahesh', 'executable': '/usr/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 2598.091000000001, 'min': 1500.0, 'max': 2450.0}, 'cpu_freq_per_core': [{'current': 3112.259, 'min': 1500.0, 'max': 2450.0}, {'current': 3139.627, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2957.636, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2927.63, 'min': 1500.0, 'max': 2450.0}, {'current': 3032.467, 'min': 1500.0, 'max': 2450.0}, {'current': 2513.1, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.776, 'min': 1500.0, 'max': 2450.0}, {'current': 2962.747, 'min': 1500.0, 'max': 2450.0}, {'current': 2513.51, 'min': 1500.0, 'max': 2450.0}, {'current': 2930.609, 'min': 1500.0, 'max': 2450.0}, {'current': 2513.894, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3139.819, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2315.049, 'min': 1500.0, 'max': 2450.0}, {'current': 2943.445, 'min': 1500.0, 'max': 2450.0}, {'current': 2946.8, 'min': 1500.0, 'max': 2450.0}, {'current': 3138.533, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.464, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2509.666, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.594, 'min': 1500.0, 'max': 2450.0}, {'current': 2878.537, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2465.919, 'min': 1500.0, 'max': 2450.0}, {'current': 2850.132, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2896.371, 'min': 1500.0, 'max': 2450.0}, {'current': 2938.29, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3089.71, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2837.308, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2904.567, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 1848.821, 'min': 1500.0, 'max': 2450.0}, {'current': 2888.866, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2874.702, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3041.93, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.325, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.345, 'min': 1500.0, 'max': 2450.0}, {'current': 3129.101, 'min': 1500.0, 'max': 2450.0}, {'current': 2521.674, 'min': 1500.0, 'max': 2450.0}, {'current': 2511.944, 'min': 1500.0, 'max': 2450.0}, {'current': 2914.302, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2932.756, 'min': 1500.0, 'max': 2450.0}, {'current': 2782.023, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2509.669, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.111, 'min': 1500.0, 'max': 2450.0}, {'current': 2508.637, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2511.287, 'min': 1500.0, 'max': 2450.0}, {'current': 2416.958, 'min': 1500.0, 'max': 2450.0}, {'current': 2418.486, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3143.262, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2612.611, 'min': 1500.0, 'max': 2450.0}, {'current': 2628.226, 'min': 1500.0, 'max': 2450.0}, {'current': 2530.233, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2526.2, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.73, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.147, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2511.751, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2950.167, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.771, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2942.716, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2844.366, 'min': 1500.0, 'max': 2450.0}, {'current': 2392.553, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2393.902, 'min': 1500.0, 'max': 2450.0}, {'current': 2864.935, 'min': 1500.0, 'max': 2450.0}, {'current': 2388.018, 'min': 1500.0, 'max': 2450.0}, {'current': 2395.204, 'min': 1500.0, 'max': 2450.0}, {'current': 2891.73, 'min': 1500.0, 'max': 2450.0}], 'disk': {'/': {'total': 125.59867858886719, 'used': 0.5812721252441406}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 4, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 251.19736099243164}} 2025-09-17 21:58:18,836 INFO HandlerThread:1208566 [system_monitor.py:probe():224] Finished collecting system info 2025-09-17 21:58:18,836 INFO HandlerThread:1208566 [system_monitor.py:probe():227] Publishing system info 2025-09-17 21:58:18,836 DEBUG HandlerThread:1208566 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2025-09-17 21:58:18,838 DEBUG HandlerThread:1208566 [system_info.py:_save_pip():68] Saving pip packages done 2025-09-17 21:58:18,841 INFO HandlerThread:1208566 [system_monitor.py:probe():229] Finished publishing system info 2025-09-17 21:58:18,860 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-17 21:58:18,860 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now 2025-09-17 21:58:18,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:58:18,876 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:58:18,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:58:19,167 DEBUG SenderThread:1208566 [sender.py:send():382] send: telemetry 2025-09-17 21:58:19,355 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/5ty3yb0v-wandb-metadata.json 2025-09-17 21:58:19,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/requirements.txt 2025-09-17 21:58:19,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-metadata.json 2025-09-17 21:58:19,628 DEBUG SenderThread:1208566 [sender.py:send():382] send: use_artifact 2025-09-17 21:58:20,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:58:22,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:58:23,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:58:24,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:58:28,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:58:33,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:58:33,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:58:33,875 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:58:33,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:58:39,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:58:44,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:58:48,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:58:48,875 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:58:48,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:58:49,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:58:49,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/config.yaml 2025-09-17 21:58:54,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:58:55,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:00,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:03,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:59:03,875 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:59:03,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:59:06,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:11,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:16,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:18,726 DEBUG SystemMonitor:1208566 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-09-17 21:59:18,731 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 21:59:18,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:59:18,875 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:59:18,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:59:21,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:26,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:26,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:59:32,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:32,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:59:33,642 DEBUG SenderThread:1208566 [sender.py:send():382] send: telemetry 2025-09-17 21:59:33,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:59:33,875 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:59:33,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:59:36,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:59:37,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:40,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:59:42,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:47,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:48,737 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 21:59:48,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 21:59:48,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 21:59:48,876 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 21:59:50,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 21:59:53,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 21:59:53,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/config.yaml 2025-09-17 21:59:58,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:02,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:03,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:00:03,876 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:00:03,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:00:04,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:04,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:06,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:08,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:09,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:10,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:12,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:14,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:14,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:16,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:18,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:18,742 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:00:18,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:00:18,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:00:18,876 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:00:20,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:20,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:22,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:24,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:25,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:26,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:28,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:30,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:31,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:32,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:33,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:00:33,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:00:33,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:00:34,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:36,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:36,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:38,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:40,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:42,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:42,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:44,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:46,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:47,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:48,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:48,746 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:00:48,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:00:48,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:00:48,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:00:50,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:52,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:53,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:54,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:56,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:00:58,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:00:58,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:00,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:02,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:03,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:01:03,876 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:01:03,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:01:04,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:04,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:06,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:08,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:09,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:10,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:12,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:14,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:15,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:16,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:18,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:18,750 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:01:18,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:01:18,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:01:18,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:01:20,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:20,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:22,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:24,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:26,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:26,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:28,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:30,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:31,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:32,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:33,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:01:33,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:01:33,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:01:34,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:36,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:36,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:38,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:40,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:42,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:42,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:44,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:46,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:48,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:48,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:48,753 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:01:48,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:01:48,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:01:48,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:01:50,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:52,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:53,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:01:54,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:56,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:58,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:01:58,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:00,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:02,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:03,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:02:03,876 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:02:03,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:02:04,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:04,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:06,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:08,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:09,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:10,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:12,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:14,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:15,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:16,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:18,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:18,797 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:02:18,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:02:18,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:02:18,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:02:20,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:20,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:22,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:24,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:26,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:26,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:28,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:30,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:31,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:32,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:33,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:02:33,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:02:33,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:02:34,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:37,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:38,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:40,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:42,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:42,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:44,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:46,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:48,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:48,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:48,800 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:02:48,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:02:48,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:02:48,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:02:50,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:52,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:53,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:02:54,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:56,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:58,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:02:59,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:00,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:02,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:03,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:03:03,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:03:03,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:03:04,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:05,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:06,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:08,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:10,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:10,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:12,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:14,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:15,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:16,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:18,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:18,802 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:03:18,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:03:18,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:03:18,878 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:03:20,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:21,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:22,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:24,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:26,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:26,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:28,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:30,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:32,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:32,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:33,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:03:33,878 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:03:33,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:03:34,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:36,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:37,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:38,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:40,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:42,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:43,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:44,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:46,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:48,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:48,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:48,807 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:03:48,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:03:48,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:03:48,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:03:50,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:52,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:54,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:03:54,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:56,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:58,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:03:59,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:00,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:02,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:03,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:04:03,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:04:03,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:04:04,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:05,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:06,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:08,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:10,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:10,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:12,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:14,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:16,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:16,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:18,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:18,809 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:04:18,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:04:18,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:04:18,878 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:04:20,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:21,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:22,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:24,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:26,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:28,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:30,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:32,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:32,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:33,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:04:33,877 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:04:33,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:04:34,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:36,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:38,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:38,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:40,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:42,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:43,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:44,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:46,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:48,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:48,811 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:04:48,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:48,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:04:48,878 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:04:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:04:50,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:52,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:54,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:04:54,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:56,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:04:58,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:00,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:00,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:02,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:03,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:05:03,878 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:05:03,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:05:04,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:05,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:06,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:08,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:10,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:11,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:12,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:14,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:16,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:16,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:18,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:18,814 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:05:18,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:05:18,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:05:18,878 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:05:20,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:22,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:22,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:24,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:26,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:27,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:28,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:30,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:32,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:33,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:33,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:05:33,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:05:33,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:05:34,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:36,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:38,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:38,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:40,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:42,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:44,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:44,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:46,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:48,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:48,820 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:05:48,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:05:48,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:05:48,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:05:49,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:50,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:52,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:54,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:55,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:05:56,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:05:58,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:00,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:00,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:02,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:03,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:06:03,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:06:03,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:06:04,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:06,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:06,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:08,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:10,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:11,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:12,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:14,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:16,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:17,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:18,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:18,822 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:06:18,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:06:18,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:06:18,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:06:20,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:22,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:22,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:24,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:26,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:28,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:28,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:30,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:32,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:33,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:33,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:06:33,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:06:33,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:06:34,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:36,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:38,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:39,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:40,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:42,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:44,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:44,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:46,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:48,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:48,824 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:06:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:06:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:06:48,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:06:50,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:50,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:52,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:54,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:55,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:06:56,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:06:58,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:00,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:00,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:02,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:03,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:07:03,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:07:03,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:07:04,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:06,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:06,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:08,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:10,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:11,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:12,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:14,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:16,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:17,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:18,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:18,827 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:07:18,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:07:18,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:07:18,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:07:20,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:22,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:22,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:24,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:26,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:28,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:28,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:30,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:32,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:33,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:07:33,879 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:07:33,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:07:34,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:34,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:36,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:38,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:39,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:40,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:42,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:44,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:44,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:46,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:48,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:48,832 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:07:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:07:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:07:48,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:07:50,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:50,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:52,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:54,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:55,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:07:56,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:07:58,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:00,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:01,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:02,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:03,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:08:03,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:08:03,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:08:04,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:06,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:06,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:08,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:10,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:12,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:12,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:14,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:16,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:17,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:18,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:18,833 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:08:18,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:08:18,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:08:18,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:08:20,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:22,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:23,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:24,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:26,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:28,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:28,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:30,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:32,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:33,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:08:33,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:08:33,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:08:34,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:34,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:36,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:38,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:39,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:40,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:42,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:44,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:45,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:46,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:48,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:48,836 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:08:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:08:48,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:08:48,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:08:50,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:50,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:52,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:54,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:56,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:08:56,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:08:58,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:00,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:01,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:02,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:03,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:09:03,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:09:03,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:09:04,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:06,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:07,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:08,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:10,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:12,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:12,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:14,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:16,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:18,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:18,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:18,839 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:09:18,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:09:18,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:09:18,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:09:20,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:22,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:23,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:24,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:26,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:28,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:29,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:30,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:32,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:33,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:09:33,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:09:33,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:09:34,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:35,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:36,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:38,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:40,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:40,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:42,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:44,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:45,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:46,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:48,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:48,844 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:09:48,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:09:48,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:09:48,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:09:50,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:51,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:52,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:54,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:56,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:09:56,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:09:58,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:00,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:02,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:02,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:03,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:10:03,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:10:03,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:10:04,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:06,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:07,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:08,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:10,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:12,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:13,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:14,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:16,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:18,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:18,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:18,844 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:10:18,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:10:18,880 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:10:18,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:10:20,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:22,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:24,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:24,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:26,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:28,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:29,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:30,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:32,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:33,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:10:33,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:10:33,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:10:34,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:35,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:36,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:38,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:40,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:40,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:42,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:44,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:46,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:46,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:48,847 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:10:48,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:48,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:10:48,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:10:48,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:10:50,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:51,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:52,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:54,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:56,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:10:57,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:10:58,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:00,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:02,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:02,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:03,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:11:03,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:11:03,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:11:04,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:06,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:08,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:08,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:10,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:12,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:13,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:14,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:16,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:18,850 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:11:18,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:18,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:11:18,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:11:18,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:11:19,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:20,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:22,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:24,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:24,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:26,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:28,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:30,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:30,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:32,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:33,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:11:33,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:11:33,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:11:34,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:35,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:36,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:38,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:40,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:41,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:42,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:44,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:46,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:46,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:48,853 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:11:48,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:48,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:11:48,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:11:48,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:11:50,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:52,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:52,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:54,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:56,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:11:57,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:11:58,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:00,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:02,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:03,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:12:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:12:03,881 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:12:04,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:06,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:08,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:08,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:10,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:12,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:14,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:14,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:16,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:18,855 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:12:18,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:12:18,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:12:18,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:12:18,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:20,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:20,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:22,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:24,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:25,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:26,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:28,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:30,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:30,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:32,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:33,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:12:33,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:12:33,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:12:34,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:36,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:36,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:38,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:40,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:41,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:42,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:44,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:46,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:47,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:48,858 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:12:48,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:12:48,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:12:48,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:12:48,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:50,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:52,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:52,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:54,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:56,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:12:58,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:12:58,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:00,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:02,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:03,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:13:03,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:13:03,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:13:04,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:06,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:08,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:09,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:10,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:12,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:14,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:14,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:16,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:18,861 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:13:18,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:13:18,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:13:18,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:13:18,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:20,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:20,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:22,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:24,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:25,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:26,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:28,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:30,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:31,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:32,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:33,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:13:33,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:13:33,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:13:34,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:36,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:36,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:38,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:40,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:42,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:42,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:44,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:46,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:47,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:48,864 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:13:48,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:13:48,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:13:48,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:13:48,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:50,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:52,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:53,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:54,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:56,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:13:58,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:13:58,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:00,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:02,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:14:03,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:14:03,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:14:04,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:04,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:06,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:08,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:09,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:10,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:12,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:14,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:15,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:16,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:18,866 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:14:18,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:14:18,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:14:18,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:14:18,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:20,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:20,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:22,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:24,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:26,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:26,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:28,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:30,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:31,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:32,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:33,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:14:33,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:14:33,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:14:34,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:36,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:37,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:38,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:40,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:42,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:42,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:44,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:46,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:48,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:48,869 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:14:48,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:14:48,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:14:48,883 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:14:48,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:50,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:52,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:53,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:14:54,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:56,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:58,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:14:59,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:00,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:02,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:03,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:15:03,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:15:03,883 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:15:04,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:04,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:06,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:08,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:10,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:10,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:12,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:14,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:15,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:16,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:18,873 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:15:18,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:15:18,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:15:18,883 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:15:18,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:20,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:21,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:22,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:24,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:26,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:26,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:28,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:30,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:32,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:32,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:33,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:15:33,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:15:33,883 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:15:34,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:36,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:37,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:38,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:40,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:42,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:43,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:44,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:46,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:48,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:48,875 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:15:48,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:15:48,883 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:15:48,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:15:48,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:50,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:52,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:54,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:15:54,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:56,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:58,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:15:59,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:00,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:02,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:03,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:16:03,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:16:03,884 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:16:04,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:05,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:06,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:08,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:10,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:10,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:12,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:14,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:15,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:16,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:18,878 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:16:18,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:16:18,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:16:18,884 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:16:18,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:21,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:21,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:23,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:25,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:26,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:27,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:29,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:31,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:32,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:33,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:33,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:16:33,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:16:33,884 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:16:35,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:37,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:37,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:39,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:41,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:43,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:43,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:45,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:47,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:48,881 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:16:48,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:48,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:16:48,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:16:48,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:16:49,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:51,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:53,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:54,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:16:55,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:57,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:59,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:16:59,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:01,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:03,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:03,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:17:03,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:17:03,884 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:17:05,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:05,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:07,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:09,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:10,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:11,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:13,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:15,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:16,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:17,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:18,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:17:18,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:17:18,887 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:17:18,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:17:19,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:21,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:21,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:23,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:25,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:27,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:27,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:29,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:31,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:32,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:33,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:33,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:17:33,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:17:33,884 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:17:35,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:37,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:38,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:39,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:41,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:43,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:43,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:45,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:47,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:17:48,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:17:48,888 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:17:48,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:17:49,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:49,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:51,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:53,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:54,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:17:55,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:57,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:17:59,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:00,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:01,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:03,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:03,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:18:03,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:18:03,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:18:05,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:08,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:10,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:10,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:12,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:14,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:16,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:16,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:18,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:18,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:18:18,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:18:18,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:18:19,088 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:18:20,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:21,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:22,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:24,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:26,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:27,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:28,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:30,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:32,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:32,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:33,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:18:33,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:18:33,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:18:34,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:36,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:38,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:38,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:40,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:42,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:43,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:44,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:46,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:48,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:48,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:18:48,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:18:48,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:18:49,009 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:18:49,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:50,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:52,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:54,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:54,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:18:56,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:18:58,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:00,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:00,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:02,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:03,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:19:03,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:19:03,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:19:04,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:06,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:06,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:08,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:10,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:11,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:12,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:14,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:16,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:17,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:18,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:18,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:19:18,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:19:18,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:19:19,008 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:19:20,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:22,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:22,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:24,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:26,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:28,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:28,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:30,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:32,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:33,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:33,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:19:33,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:19:33,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:19:34,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:36,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:38,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:39,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:40,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:42,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:44,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:44,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:46,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:48,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:48,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:19:48,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:19:48,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:19:49,215 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:19:50,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:50,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:52,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:54,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:55,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:19:56,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:19:58,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:00,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:01,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:02,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:03,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:20:03,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:20:03,885 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:20:04,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:06,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:06,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:08,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:10,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:12,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:12,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:14,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:16,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:17,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:18,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:18,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:20:18,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:20:18,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:20:19,031 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:20:20,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:22,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:23,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:24,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:26,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:28,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:28,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:30,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:32,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:33,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:20:33,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:20:33,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:20:34,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:34,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:36,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:38,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:39,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:40,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:42,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:44,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:44,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:46,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:48,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:48,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:20:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:20:48,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:20:49,104 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:20:50,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:50,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:52,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:54,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:55,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:20:56,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:20:58,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:00,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:01,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:02,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:03,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:21:03,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:21:03,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:21:04,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:06,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:06,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:08,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:10,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:12,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:12,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:14,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:16,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:17,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:18,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:18,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:21:18,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:21:18,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:21:19,027 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:21:20,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:22,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:23,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:24,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:26,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:28,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:28,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:30,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:32,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:33,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:21:33,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:21:33,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:21:33,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:34,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:36,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:38,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:39,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:40,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:42,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:44,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:45,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:46,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:48,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:21:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:21:48,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:21:49,018 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:21:50,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:50,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:52,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:54,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:56,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:21:56,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:21:58,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:00,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:01,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:02,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:03,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:22:03,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:22:03,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:22:04,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:06,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:07,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:08,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:10,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:12,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:12,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:14,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:16,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:18,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:18,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:18,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:22:18,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:22:18,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:22:19,037 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:22:20,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:22,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:23,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:24,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:26,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:28,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:29,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:30,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:32,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:33,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:22:33,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:22:33,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:22:34,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:35,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:36,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:38,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:40,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:40,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:42,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:44,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:45,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:46,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:48,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:22:48,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:22:48,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:22:49,068 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:22:50,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:51,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:52,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:54,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:56,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:22:56,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:22:58,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:00,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:02,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:02,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:03,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:03,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:23:03,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:23:03,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:23:04,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:04,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:04,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:23:06,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:07,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:08,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:10,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:12,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:12,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:14,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:16,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:18,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:18,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:18,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:23:18,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:23:18,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:23:19,061 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:23:20,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:22,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:23,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:24,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:26,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:28,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:29,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:30,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:32,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:33,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:23:33,886 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:23:33,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:23:34,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:35,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:36,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:38,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:40,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:40,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:42,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:44,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:45,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:46,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:48,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:23:48,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:23:48,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:23:49,010 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:23:50,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:51,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:52,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:54,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:56,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:23:56,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:23:58,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:00,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:02,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:02,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:03,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:24:03,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:24:03,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:24:04,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:06,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:07,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:08,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:10,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:12,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:13,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:14,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:16,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:18,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:18,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:18,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:24:18,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:24:18,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:24:19,129 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:24:20,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:22,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:24,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:24,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:26,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:28,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:29,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:30,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:32,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:33,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:24:33,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:24:33,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:24:34,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:35,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:36,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:38,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:40,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:40,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:42,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:44,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:46,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:46,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:48,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:48,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:24:48,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:24:48,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:24:49,046 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:24:50,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:51,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:52,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:54,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:56,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:24:57,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:24:58,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:00,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:02,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:02,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:03,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:25:03,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:25:03,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:25:04,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:06,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:08,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:08,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:10,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:12,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:13,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:14,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:16,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:18,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:18,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:25:18,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:25:18,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:25:19,031 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:25:19,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:20,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:22,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:24,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:24,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:26,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:28,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:30,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:30,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:32,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:33,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:25:33,887 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:25:33,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:25:34,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:35,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:36,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:38,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:40,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:41,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:42,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:44,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:46,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:46,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:48,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:48,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:25:48,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:25:48,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:25:49,028 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:25:50,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:52,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:52,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:54,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:56,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:25:57,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:25:58,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:00,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:02,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:03,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:03,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:26:03,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:26:03,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:26:04,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:06,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:08,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:08,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:10,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:12,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:14,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:14,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:16,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:18,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:18,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:26:18,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:26:18,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:26:19,034 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:26:20,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:20,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:22,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:24,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:25,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:26,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:28,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:30,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:30,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:32,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:33,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:26:33,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:26:33,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:26:34,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:36,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:36,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:38,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:40,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:41,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:42,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:44,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:46,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:47,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:48,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:48,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:26:48,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:26:48,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:26:48,996 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:26:50,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:52,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:52,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:54,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:56,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:26:58,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:26:58,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:00,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:02,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:03,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:03,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:27:03,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:27:03,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:27:04,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:06,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:08,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:09,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:10,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:12,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:14,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:14,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:16,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:18,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:18,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:27:18,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:27:18,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:27:19,085 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:27:20,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:20,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:22,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:24,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:25,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:26,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:28,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:30,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:31,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:32,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:33,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:27:33,888 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:27:33,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:27:34,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:36,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:36,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:38,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:40,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:42,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:42,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:44,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:46,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:47,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:48,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:48,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:27:48,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:27:48,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:27:49,016 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:27:50,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:52,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:53,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:27:54,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:56,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:58,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:27:58,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:00,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:02,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:03,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:28:03,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:28:03,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:28:04,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:04,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:06,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:08,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:09,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:10,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:12,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:14,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:15,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:16,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:18,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:18,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:28:18,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:28:18,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:28:19,002 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:28:20,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:20,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:22,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:24,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:26,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:26,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:28,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:30,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:31,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:32,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:33,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:28:33,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:28:33,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:28:34,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:36,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:37,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:38,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:40,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:42,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:42,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:44,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:46,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:48,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:48,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:48,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:28:48,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:28:48,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:28:49,043 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:28:50,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:52,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:53,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:28:54,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:56,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:58,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:28:59,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:00,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:02,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:03,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:29:03,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:29:03,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:29:04,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:05,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:06,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:08,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:10,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:10,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:12,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:14,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:15,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:16,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:18,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:18,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:29:18,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:29:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:29:19,051 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:29:20,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:21,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:22,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:24,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:26,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:26,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:28,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:30,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:32,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:32,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:33,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:29:33,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:29:33,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:29:34,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:36,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:37,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:38,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:40,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:42,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:43,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:44,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:46,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:48,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:48,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:48,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:29:48,889 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:29:48,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:29:49,088 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:29:50,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:52,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:54,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:29:54,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:56,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:58,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:29:59,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:00,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:02,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:03,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:30:03,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:30:03,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:30:04,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:05,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:06,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:08,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:10,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:10,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:12,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:14,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:16,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:16,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:18,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:18,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:30:18,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:30:18,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:30:19,018 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:30:20,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:21,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:22,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:24,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:26,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:27,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:28,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:30,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:32,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:32,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:33,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:30:33,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:30:33,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:30:34,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:36,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:38,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:38,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:40,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:42,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:43,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:44,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:46,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:48,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:48,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:30:48,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:30:48,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:30:49,086 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:30:49,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:50,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:52,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:54,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:54,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:30:56,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:30:58,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:00,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:00,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:02,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:03,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:31:03,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:31:03,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:31:04,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:05,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:06,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:08,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:10,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:11,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:12,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:14,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:16,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:16,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:18,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:18,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:31:18,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:31:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:31:19,097 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:31:20,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:22,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:24,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:26,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:27,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:28,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:30,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:32,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:32,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:33,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:31:33,890 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:31:33,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:31:34,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:36,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:38,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:38,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:40,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:42,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:43,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:44,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:46,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:48,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:48,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:31:48,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:31:48,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:31:49,047 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:31:49,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:50,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:52,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:54,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:54,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:31:56,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:31:58,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:00,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:00,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:02,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:03,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:32:03,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:32:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:32:04,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:05,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:06,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:08,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:10,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:11,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:12,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:14,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:16,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:16,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:18,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:18,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:32:18,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:32:18,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:32:19,092 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:32:20,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:22,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:22,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:24,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:26,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:27,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:28,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:30,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:32,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:33,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:33,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:32:33,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:32:33,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:32:34,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:36,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:38,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:38,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:40,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:42,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:44,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:44,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:46,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:48,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:48,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:32:48,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:32:48,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:32:49,032 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:32:50,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:50,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:55,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:32:56,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:32:58,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:00,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:00,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:02,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:03,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:33:03,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:33:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:33:04,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:05,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:06,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:08,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:10,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:11,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:12,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:14,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:16,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:16,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:18,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:18,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:33:18,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:33:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:33:19,055 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:33:20,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:22,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:22,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:24,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:26,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:27,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:28,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:30,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:32,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:33,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:33,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:33:33,891 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:33:33,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:33:34,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:36,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:38,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:38,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:40,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:42,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:44,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:44,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:46,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:48,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:48,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:33:48,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:33:48,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:33:49,014 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:33:50,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:50,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:52,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:54,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:55,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:33:56,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:33:58,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:00,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:00,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:02,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:03,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:34:03,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:34:03,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:34:04,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:06,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:06,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:08,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:10,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:11,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:12,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:14,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:16,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:17,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:18,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:18,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:34:18,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:34:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:34:19,105 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:34:20,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:22,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:22,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:24,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:26,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:28,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:28,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:30,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:32,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:33,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:33,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:34:33,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:34:33,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:34:34,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:36,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:38,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:39,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:40,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:42,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:44,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:44,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:46,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:48,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:48,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:34:48,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:34:48,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:34:49,051 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:34:50,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:50,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:52,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:54,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:55,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:34:56,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:34:58,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:00,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:01,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:02,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:03,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:35:03,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:35:03,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:35:04,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:06,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:06,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:08,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:10,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:12,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:12,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:14,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:16,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:17,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:18,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:35:18,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:35:18,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:35:19,073 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:35:20,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:22,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:23,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:24,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:26,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:28,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:28,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:30,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:32,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:33,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:35:33,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:35:33,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:35:34,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:34,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:36,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:38,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:39,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:40,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:42,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:44,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:45,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:46,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:48,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:48,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:35:48,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:35:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:35:49,059 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:35:50,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:50,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:52,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:54,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:56,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:35:56,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:35:58,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:00,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:01,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:02,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:03,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:36:03,892 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:36:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:36:04,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:06,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:07,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:08,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:10,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:12,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:12,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:14,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:16,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:18,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:18,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:36:18,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:36:18,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:36:19,119 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:36:20,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:22,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:23,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:24,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:26,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:28,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:29,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:30,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:32,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:33,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:36:33,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:36:33,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:36:34,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:35,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:36,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:38,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:40,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:40,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:42,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:44,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:45,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:46,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:48,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:48,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:36:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:36:48,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:36:49,013 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:36:50,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:51,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:52,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:54,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:56,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:36:56,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:36:58,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:00,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:02,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:02,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:37:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:37:03,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:37:04,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:06,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:07,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:08,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:10,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:12,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:13,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:14,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:16,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:18,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:18,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:18,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:37:18,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:37:18,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:37:19,061 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:37:20,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:22,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:24,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:24,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:26,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:28,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:29,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:30,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:32,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:33,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:37:33,893 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:37:33,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:37:34,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:35,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:36,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:38,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:40,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:40,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:42,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:44,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:46,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:46,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:48,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:37:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:37:48,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:37:49,093 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:37:50,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:51,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:52,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:54,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:56,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:37:57,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:37:58,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:00,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:02,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:02,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:38:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:38:03,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:38:04,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:06,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:08,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:08,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:10,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:12,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:13,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:14,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:16,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:18,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:18,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:38:18,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:38:18,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:38:19,082 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:38:19,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:20,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:22,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:24,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:24,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:26,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:28,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:29,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:30,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:32,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:33,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:38:33,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:38:33,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:38:34,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:35,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:36,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:38,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:40,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:40,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:42,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:44,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:46,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:46,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:48,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:38:48,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:38:48,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:38:49,081 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:38:50,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:51,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:52,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:54,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:56,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:38:56,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:38:58,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:00,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:02,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:02,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:03,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:39:03,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:39:03,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:39:04,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:06,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:07,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:08,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:10,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:12,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:12,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:14,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:16,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:18,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:18,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:18,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:39:18,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:39:18,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:39:19,111 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:39:20,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:22,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:23,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:24,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:26,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:28,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:28,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:30,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:32,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:33,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:39:33,894 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:39:33,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:39:34,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:34,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:35,445 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-17 22:39:35,445 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/videos/Inference samples_0_07b9411d7e9b5ed6f3a1.gif with policy now 2025-09-17 22:39:35,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:39:35,467 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-17 22:39:35,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-17 22:39:35,483 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-17 22:39:35,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-17 22:39:35,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/videos/Inference samples_0_07b9411d7e9b5ed6f3a1.gif 2025-09-17 22:39:35,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/videos 2025-09-17 22:39:35,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media 2025-09-17 22:39:36,060 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/8a7jj7jz-media/videos/Inference samples_0_07b9411d7e9b5ed6f3a1.gif 2025-09-17 22:39:36,436 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/is5h9h3f-media/videos/Inference samples_0_07b9411d7e9b5ed6f3a1.gif 2025-09-17 22:39:36,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:36,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:39:38,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:39,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:44,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:48,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:49,012 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:39:49,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:39:49,281 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:39:50,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:50,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:39:55,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:39:57,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:39:57,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:39:57,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-17 22:39:57,684 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-17 22:39:57,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:39:58,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:39:58,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_1_83148a7870823ece0187.table.json 2025-09-17 22:39:58,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media 2025-09-17 22:39:58,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-17 22:39:58,684 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmp7kts3uyj 2025-09-17 22:39:58,908 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/wfwpkbel-media/table/rollouts_1_83148a7870823ece0187.table.json 2025-09-17 22:39:59,524 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTIyMTM0NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': None}} 2025-09-17 22:39:59,524 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-17 22:39:59,524 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_1_83148a7870823ece0187.table.json with policy now 2025-09-17 22:39:59,857 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/m35vn3j2-media/table/rollouts_1_83148a7870823ece0187.table.json 2025-09-17 22:40:00,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:00,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:02,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:03,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:40:03,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:40:03,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:40:04,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:05,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:06,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:08,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:10,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:10,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:12,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:14,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:16,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:16,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:18,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:18,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:40:18,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:40:18,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:40:19,109 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:40:20,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:21,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:22,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:24,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:26,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:27,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:28,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:30,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:32,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:32,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:33,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:40:33,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:40:33,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:40:34,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:36,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:38,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:38,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:40,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:42,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:43,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:44,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:46,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:48,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:48,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:40:48,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:40:48,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:40:49,090 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:40:49,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:50,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:52,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:54,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:54,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:40:56,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:40:58,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:00,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:00,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:02,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:03,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:41:03,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:41:03,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:41:04,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:05,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:06,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:08,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:10,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:11,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:12,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:14,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:16,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:16,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:18,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:18,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:41:18,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:41:18,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:41:19,149 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:41:20,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:22,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:22,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:24,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:26,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:27,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:28,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:30,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:32,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:33,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:33,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:41:33,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:41:33,913 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:41:34,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:36,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:38,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:39,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:40,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:42,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:44,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:44,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:46,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:48,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:48,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:41:48,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:41:48,914 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:41:49,128 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:41:50,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:50,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:52,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:55,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:41:55,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:57,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:41:59,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:01,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:01,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:03,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:03,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:42:03,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:42:03,914 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:42:05,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:06,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:07,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:09,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:11,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:12,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:13,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:15,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:17,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:17,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:18,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:42:18,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:42:18,914 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:42:19,089 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:42:19,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:21,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:23,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:23,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:25,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:27,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:28,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:29,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:31,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:33,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:33,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:42:33,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:42:33,914 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:42:34,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:35,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:37,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:39,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:39,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:41,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:43,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:45,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:45,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:47,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:48,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:42:48,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:42:48,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:42:49,058 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:42:49,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:50,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:51,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:53,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:55,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:56,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:42:57,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:42:59,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:01,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:01,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:03,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:03,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:43:03,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:43:03,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:43:05,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:07,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:07,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:09,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:11,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:12,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:13,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:15,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:17,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:18,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:18,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:43:18,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:43:18,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:43:19,060 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:43:19,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:21,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:23,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:23,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:25,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:27,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:29,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:29,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:31,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:33,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:33,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:43:33,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:43:33,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:43:35,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:35,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:37,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:39,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:40,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:41,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:43,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:45,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:45,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:47,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:48,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:43:48,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:43:48,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:43:49,063 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:43:49,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:51,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:51,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:53,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:55,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:56,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:43:57,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:43:59,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:01,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:02,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:03,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:03,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:44:03,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:44:03,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:44:05,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:07,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:08,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:09,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:11,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:13,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:13,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:15,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:17,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:18,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:44:18,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:44:18,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:44:19,099 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:44:19,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:19,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:21,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:23,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:24,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:25,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:27,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:29,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:30,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:31,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:33,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:33,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:44:33,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:44:33,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:44:35,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:35,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:37,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:39,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:41,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:41,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:43,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:45,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:46,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:47,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:48,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:44:48,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:44:48,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:44:49,072 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:44:49,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:51,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:52,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:53,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:55,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:57,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:44:57,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:44:59,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:01,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:03,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:03,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:03,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:45:03,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:45:03,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:45:05,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:07,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:08,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:09,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:11,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:13,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:14,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:15,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:17,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:18,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:45:18,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:45:18,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:45:19,071 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:45:19,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:20,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:21,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:23,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:25,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:27,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:29,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:30,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:31,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:33,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:33,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:45:33,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:45:33,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:45:35,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:36,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:37,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:39,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:41,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:41,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:43,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:45,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:47,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:47,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:48,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:45:48,915 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:45:48,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:45:49,074 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:45:49,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:51,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:52,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:53,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:55,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:57,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:45:58,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:45:59,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:01,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:03,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:03,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:46:03,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:46:03,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:46:04,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:05,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:07,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:09,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:09,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:11,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:13,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:14,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:15,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:17,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:18,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:46:18,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:46:18,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:46:19,107 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:46:19,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:20,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:21,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:23,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:25,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:25,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:27,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:29,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:31,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:31,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:33,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:46:33,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:46:33,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:46:33,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:35,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:37,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:37,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:39,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:41,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:42,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:43,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:45,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:47,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:48,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:48,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:46:48,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:46:48,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:46:49,081 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:46:49,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:51,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:53,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:53,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:55,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:57,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:46:59,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:46:59,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:01,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:03,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:47:03,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:47:03,917 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:47:03,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:04,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:05,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:07,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:09,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:10,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:10,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:10,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:11,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:11,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:11,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 22:47:11,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:13,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:15,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:15,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:17,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:18,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:47:18,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:47:18,916 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:47:19,083 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:47:19,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:21,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:21,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:23,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:25,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:26,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:27,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:29,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:31,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:32,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:33,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:47:33,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:47:33,917 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:47:33,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:35,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:37,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:37,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:39,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:41,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:43,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:43,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:45,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:47,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:48,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:47:48,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:47:48,917 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:47:49,113 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:47:49,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:49,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:51,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:53,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:54,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:55,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:57,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:47:59,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:47:59,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:01,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:03,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:48:03,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:48:03,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:48:03,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:05,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:05,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:07,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:09,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:11,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:11,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:13,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:15,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:16,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:17,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:18,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:48:18,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:48:18,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:48:19,091 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:48:19,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:21,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:22,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:24,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:26,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:27,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:28,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:30,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:32,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:33,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:33,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:48:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:48:33,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:48:34,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:36,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:38,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:38,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:40,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:42,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:44,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:44,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:46,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:48,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:48,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:48:48,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:48:48,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:48:49,124 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:48:49,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:50,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:52,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:54,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:54,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:48:56,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:58,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:48:59,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:00,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:02,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:03,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:49:03,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:49:03,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:49:04,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:05,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:06,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:08,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:10,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:10,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:12,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:14,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:15,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:16,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:18,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:18,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:49:18,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:49:18,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:49:19,096 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:49:20,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:21,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:22,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:24,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:26,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:26,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:28,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:30,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:32,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:32,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:33,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:49:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:49:33,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:49:34,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:36,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:37,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:38,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:40,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:42,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:43,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:44,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:46,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:48,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:48,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:49:48,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:49:48,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:49:49,957 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:49:49,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:50,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:52,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:54,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:55,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:49:56,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:49:58,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:00,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:01,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:02,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:03,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:50:03,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:50:03,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:50:04,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:06,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:06,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:08,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:10,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:12,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:12,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:14,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:16,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:17,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:18,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:18,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:50:18,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:50:18,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:50:19,106 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:50:20,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:22,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:23,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:24,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:26,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:28,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:28,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:30,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:32,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:33,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:50:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:50:33,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:50:34,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:34,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:36,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:38,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:40,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:40,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:42,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:44,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:45,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:46,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:48,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:48,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:50:48,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:50:48,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:50:49,111 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:50:50,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:51,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:52,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:54,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:56,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:50:56,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:50:58,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:00,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:02,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:02,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:03,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:51:03,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:51:03,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:51:04,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:06,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:07,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:08,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:10,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:12,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:13,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:14,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:16,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:18,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:18,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:18,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:51:18,918 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:51:18,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:51:19,107 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:51:20,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:22,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:24,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:24,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:26,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:28,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:29,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:30,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:32,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:51:33,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:51:33,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:51:34,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:35,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:36,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:38,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:40,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:40,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:42,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:44,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:46,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:46,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:48,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:48,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:51:48,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:51:48,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:51:49,109 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:51:50,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:51,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:52,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:54,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:56,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:51:57,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:51:58,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:00,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:02,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:02,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:03,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:52:03,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:52:03,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:52:04,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:06,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:08,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:08,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:10,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:12,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:13,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:14,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:16,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:18,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:18,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:52:18,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:52:18,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:52:19,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:19,114 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:52:20,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:22,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:24,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:24,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:26,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:28,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:30,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:30,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:32,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:33,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:52:33,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:52:33,920 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:52:34,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:35,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:36,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:38,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:40,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:41,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:42,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:44,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:46,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:46,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:48,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:48,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:52:48,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:52:48,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:52:49,116 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:52:50,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:52,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:52,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:54,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:56,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:52:57,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:52:58,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:00,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:02,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:03,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:03,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:53:03,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:53:03,919 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:53:04,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:06,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:08,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:08,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:10,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:12,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:14,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:14,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:16,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:18,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:18,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:53:18,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:53:18,920 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:53:19,118 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:53:20,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:20,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:22,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:24,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:25,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:26,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:28,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:30,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:31,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:32,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:33,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:53:33,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:53:33,920 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:53:34,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:36,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:36,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:38,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:40,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:42,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:42,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:44,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:46,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:47,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:48,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:48,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:53:48,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:53:48,920 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:53:49,121 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:53:50,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:52,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:53,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:53:54,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:56,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:58,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:53:58,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:00,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:02,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:03,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:54:03,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:54:03,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:54:04,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:04,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:06,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:08,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:09,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:10,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:12,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:14,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:15,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:16,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:18,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:18,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:54:18,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:54:18,920 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:54:19,123 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:54:20,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:20,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:22,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:24,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:26,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:26,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:28,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:30,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:31,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:32,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:33,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:54:33,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:54:33,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:54:34,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:36,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:37,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:38,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:40,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:42,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:42,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:44,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:46,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:48,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:48,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:48,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:54:48,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:54:48,920 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:54:49,149 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:54:50,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:52,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:53,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:54:54,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:56,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:58,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:54:59,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:00,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:02,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:03,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:55:03,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:55:03,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:55:04,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:05,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:06,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:08,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:10,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:10,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:12,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:14,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:15,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:16,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:18,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:18,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:55:18,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:55:18,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:55:19,129 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:55:20,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:21,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:22,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:24,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:26,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:26,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:28,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:30,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:32,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:32,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:33,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:55:33,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:55:33,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:55:34,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:36,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:37,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:38,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:40,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:42,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:43,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:44,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:46,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:48,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:48,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:55:48,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:55:48,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:55:49,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:49,131 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:55:50,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:52,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:54,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:54,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:55:56,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:58,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:55:59,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:00,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:02,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:03,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:56:03,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:56:03,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:56:04,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:05,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:06,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:08,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:10,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:11,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:12,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:14,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:16,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:16,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:18,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:18,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:56:18,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:56:18,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:56:19,139 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:56:20,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:22,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:22,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:24,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:26,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:27,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:28,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:30,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:32,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:33,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:33,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:56:33,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:56:33,921 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:56:34,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:36,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:38,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:38,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:40,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:42,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:44,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:44,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:46,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:48,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:48,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:56:48,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:56:48,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:56:49,137 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:56:49,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:50,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:52,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:54,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:55,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:56:56,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:56:58,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:00,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:00,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:02,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:03,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:57:03,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:57:03,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:57:04,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:06,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:06,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:08,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:10,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:11,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:12,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:14,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:16,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:17,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:18,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:18,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:57:18,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:57:18,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:57:19,140 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:57:20,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:22,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:22,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:24,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:26,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:28,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:28,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:30,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:32,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:33,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:33,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:57:33,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:57:33,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:57:34,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:36,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:38,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:39,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:40,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:42,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:44,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:44,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:46,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:48,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:48,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:57:48,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:57:48,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:57:49,142 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:57:50,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:50,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:52,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:54,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:55,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:57:56,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:57:58,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:00,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:01,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:02,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:03,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:58:03,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:58:03,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:58:04,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:06,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:06,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:08,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:10,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:12,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:12,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:14,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:16,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:17,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:18,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:18,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:58:18,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:58:18,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:58:19,145 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:58:20,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:22,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:23,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:24,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:26,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:28,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:28,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:30,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:32,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:33,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:58:33,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:58:33,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:58:34,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:34,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:36,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:38,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:39,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:40,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:42,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:44,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:45,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:46,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:48,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:48,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:58:48,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:58:48,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:58:49,148 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:58:50,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:51,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:52,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:54,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:56,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:58:56,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:58:58,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:00,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:02,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:02,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:03,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:59:03,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:59:03,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:59:04,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:06,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:07,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:08,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:10,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:12,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:13,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:14,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:16,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:18,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:18,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:18,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:59:18,922 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:59:18,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:59:19,151 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:59:20,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:22,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:24,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:24,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:26,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:28,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:29,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:30,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:32,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:33,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:59:33,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:59:33,924 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:59:34,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:35,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:36,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:38,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:40,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:40,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:42,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:44,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:46,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:46,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:48,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:48,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 22:59:48,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 22:59:48,924 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 22:59:49,155 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 22:59:50,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:51,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:52,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:54,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:56,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 22:59:57,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 22:59:58,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:00,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:02,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:02,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:03,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:00:03,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:00:03,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:00:04,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:06,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:08,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:08,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:10,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:12,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:13,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:14,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:16,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:18,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:18,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:00:18,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:00:18,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:00:19,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:19,156 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:00:20,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:22,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:24,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:24,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:26,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:28,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:30,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:30,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:32,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:33,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:00:33,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:00:33,924 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:00:34,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:35,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:36,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:38,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:40,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:41,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:42,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:44,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:46,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:46,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:48,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:48,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:00:48,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:00:48,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:00:49,158 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:00:50,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:52,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:52,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:54,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:56,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:00:58,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:00:58,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:00,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:02,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:03,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:03,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:01:03,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:01:03,923 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:01:04,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:06,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:08,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:09,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:10,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:12,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:14,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:14,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:16,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:18,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:18,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:01:18,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:01:18,924 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:01:19,162 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:01:20,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:20,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:22,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:24,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:25,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:26,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:28,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:30,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:31,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:32,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:33,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:01:33,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:01:33,925 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:01:34,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:36,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:36,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:38,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:40,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:42,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:42,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:44,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:46,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:47,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:48,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:48,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:01:48,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:01:48,924 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:01:49,166 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:01:50,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:52,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:53,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:01:54,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:56,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:58,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:01:58,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:00,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:02,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:03,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:02:03,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:02:03,925 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:02:04,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:04,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:06,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:08,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:09,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:10,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:12,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:14,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:15,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:16,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:18,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:18,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:02:18,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:02:18,925 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:02:19,167 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:02:20,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:20,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:22,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:24,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:26,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:26,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:28,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:30,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:31,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:32,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:33,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:02:33,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:02:33,925 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:02:34,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:36,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:37,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:38,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:40,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:42,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:42,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:44,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:46,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:48,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:48,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:48,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:02:48,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:02:48,924 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:02:49,169 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:02:50,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:52,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:53,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:02:54,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:56,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:58,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:02:59,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:00,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:02,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:03,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:03:03,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:03:03,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:03:04,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:05,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:06,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:08,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:10,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:10,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:12,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:14,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:16,054 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:16,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:18,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:18,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:03:18,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:03:18,925 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:03:19,173 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:03:20,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:21,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:22,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:24,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:26,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:27,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:28,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:30,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:32,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:32,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:33,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:03:33,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:03:33,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:03:34,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:36,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:38,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:38,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:40,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:42,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:43,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:44,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:46,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:48,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:48,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:03:48,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:03:48,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:03:49,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:49,178 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:03:50,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:52,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:54,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:03:54,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:56,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:03:58,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:00,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:00,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:02,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:03,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:04:03,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:04:03,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:04:04,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:05,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:06,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:08,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:10,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:11,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:12,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:14,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:16,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:16,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:18,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:18,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:04:18,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:04:18,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:04:19,178 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:04:20,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:22,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:22,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:24,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:26,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:27,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:28,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:30,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:32,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:33,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:33,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:04:33,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:04:33,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:04:34,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:36,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:38,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:38,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:40,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:42,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:44,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:44,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:46,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:48,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:48,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:04:48,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:04:48,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:04:49,181 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:04:50,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:50,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:52,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:54,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:55,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:04:56,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:04:58,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:00,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:00,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:02,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:03,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:05:03,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:05:03,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:05:04,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:06,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:06,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:08,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:10,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:11,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:12,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:14,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:16,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:17,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:18,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:18,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:05:18,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:05:18,926 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:05:19,184 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:05:20,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:22,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:22,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:24,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:26,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:28,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:28,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:30,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:32,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:33,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:05:33,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:05:33,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:05:34,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:34,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:36,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:38,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:39,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:40,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:42,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:44,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:44,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:46,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:48,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:48,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:05:48,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:05:48,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:05:49,187 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:05:50,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:50,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:52,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:54,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:56,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:05:56,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:05:58,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:00,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:01,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:02,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:03,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:06:03,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:06:03,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:06:04,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:06,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:07,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:08,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:10,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:12,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:12,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:14,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:16,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:18,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:18,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:18,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:06:18,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:06:18,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:06:19,190 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:06:20,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:22,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:23,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:24,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:26,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:28,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:29,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:30,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:32,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:33,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:06:33,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:06:33,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:06:34,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:35,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:36,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:38,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:40,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:40,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:42,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:44,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:45,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:46,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:48,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:48,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:06:48,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:06:48,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:06:49,194 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:06:50,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:51,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:52,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:54,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:56,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:06:56,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:06:58,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:00,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:02,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:02,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:07:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:07:03,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:07:04,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:06,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:07,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:08,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:10,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:12,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:13,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:14,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:16,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:18,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:18,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:18,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:07:18,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:07:18,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:07:19,197 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:07:20,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:22,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:24,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:24,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:26,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:28,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:29,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:30,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:32,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:33,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:07:33,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:07:33,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:07:34,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:35,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:36,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:38,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:40,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:40,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:42,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:44,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:46,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:46,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:48,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:48,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:07:48,927 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:07:48,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:07:49,198 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:07:50,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:51,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:52,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:54,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:56,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:07:57,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:07:58,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:00,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:02,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:02,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:08:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:08:03,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:08:04,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:06,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:08,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:08,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:10,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:12,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:13,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:14,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:16,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:18,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:18,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:08:18,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:08:18,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:08:19,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:19,201 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:08:20,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:22,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:24,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:25,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:26,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:28,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:30,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:30,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:32,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:33,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:08:33,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:08:33,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:08:34,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:36,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:36,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:38,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:40,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:41,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:42,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:44,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:46,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:47,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:48,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:48,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:08:48,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:08:48,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:08:49,203 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:08:50,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:52,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:52,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:54,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:56,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:08:58,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:08:58,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:00,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:02,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:03,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:09:03,928 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:09:03,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:09:04,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:06,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:08,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:09,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:10,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:12,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:14,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:14,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:16,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:18,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:09:18,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:09:18,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:09:18,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:19,208 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:09:20,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:20,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:22,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:24,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:25,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:26,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:28,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:30,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:31,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:32,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:33,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:09:33,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:09:33,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:09:34,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:36,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:36,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:38,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:40,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:42,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:42,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:44,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:46,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:47,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:48,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:09:48,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:09:48,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:09:48,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:49,208 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:09:50,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:52,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:53,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:54,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:56,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:09:58,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:09:58,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:00,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:02,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:03,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:10:03,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:10:03,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:10:04,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:04,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:06,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:08,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:09,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:10,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:10:10,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:12,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:14,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:15,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:16,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:18,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:10:18,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:10:18,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:10:18,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:19,213 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:10:20,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:21,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:22,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:24,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:26,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:26,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:28,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:30,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:32,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:32,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:33,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:10:33,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:10:33,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:10:34,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:36,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:37,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:38,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:40,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:42,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:43,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:44,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:46,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:48,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:48,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:10:48,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:10:48,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:10:48,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:49,216 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:10:50,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:52,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:54,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:10:54,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:57,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:59,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:10:59,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:01,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:03,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:03,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:11:03,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:11:03,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:11:05,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:05,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:07,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:09,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:10,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:11,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:13,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:15,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:16,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:17,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:18,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:11:18,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:11:18,929 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:11:19,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:19,218 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:11:21,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:21,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:23,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:25,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:27,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:27,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:29,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:31,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:32,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:33,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:33,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:11:33,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:11:33,930 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:11:35,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:37,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:38,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:39,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:41,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:43,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:43,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:45,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:47,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:48,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:11:48,930 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:11:48,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:11:49,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:49,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:49,220 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:11:51,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:53,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:55,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:11:55,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:57,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:11:59,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:00,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:01,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:03,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:03,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:12:03,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:12:03,930 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:12:05,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:06,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:07,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:09,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:11,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:11,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:13,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:15,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:17,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:17,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:18,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:12:18,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:12:18,930 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:12:19,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:19,223 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:12:21,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:22,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:23,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:25,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:27,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:28,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:29,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:31,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:33,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:33,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:33,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:12:33,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:12:33,930 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:12:35,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:37,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:39,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:39,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:41,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:43,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:44,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:45,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:47,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:48,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:12:48,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:12:48,930 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:12:49,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:49,227 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:12:50,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:51,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:53,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:55,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:55,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:12:57,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:12:59,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:01,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:01,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:03,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:03,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:13:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:13:03,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:13:05,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:06,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:07,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:09,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:11,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:12,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:13,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:15,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:17,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:17,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:18,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:13:18,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:13:18,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:13:19,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:19,229 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:13:21,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:23,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:23,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:25,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:27,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:28,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:29,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:31,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:33,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:33,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:13:33,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:13:33,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:13:34,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:35,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:37,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:39,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:39,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:41,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:43,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:45,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:45,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:47,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:48,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:13:48,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:13:48,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:13:49,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:49,231 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:13:50,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:51,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:53,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:55,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:56,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:13:57,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:13:59,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:01,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:01,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:03,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:03,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:14:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:14:03,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:14:05,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:07,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:07,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:09,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:11,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:12,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:13,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:15,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:17,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:18,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:18,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:14:18,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:14:18,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:14:19,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:19,234 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:14:21,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:23,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:23,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:25,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:27,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:29,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:29,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:31,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:33,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:33,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:14:33,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:14:33,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:14:35,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:35,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:37,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:39,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:40,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:41,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:43,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:45,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:46,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:47,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:48,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:14:48,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:14:48,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:14:49,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:49,240 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:14:51,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:51,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:53,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:55,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:57,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:14:57,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:14:59,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:01,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:02,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:03,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:15:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:15:03,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:15:05,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:07,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:08,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:09,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:11,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:13,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:13,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:15,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:17,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:18,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:15:18,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:15:18,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:15:19,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:19,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:19,240 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:15:21,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:23,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:24,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:25,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:27,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:29,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:30,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:31,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:33,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:33,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:15:33,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:15:33,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:15:35,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:35,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:37,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:39,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:41,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:41,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:43,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:45,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:46,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:47,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:48,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:15:48,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:15:48,931 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:15:49,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:49,243 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:15:51,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:52,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:53,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:55,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:57,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:15:57,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:15:59,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:01,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:03,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:03,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:16:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:16:03,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:16:05,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:07,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:08,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:09,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:11,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:13,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:14,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:15,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:17,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:18,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:16:18,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:16:18,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:16:19,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:19,246 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:16:20,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:21,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:23,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:25,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:25,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:27,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:29,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:30,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:31,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:33,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:33,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:16:33,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:16:33,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:16:35,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:36,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:37,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:39,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:41,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:41,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:43,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:45,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:47,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:47,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:48,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:16:48,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:16:48,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:16:49,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:49,250 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:16:51,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:52,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:53,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:55,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:57,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:16:58,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:16:59,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:01,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:03,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:17:03,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:17:03,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:17:04,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:05,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:07,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:09,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:09,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:11,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:13,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:14,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:15,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:17,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:18,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:17:18,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:17:18,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:17:19,251 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:17:19,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:20,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:21,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:24,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:26,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:26,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:28,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:30,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:31,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:32,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:33,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:17:33,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:17:33,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:17:34,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:36,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:37,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:38,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:40,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:42,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:42,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:44,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:46,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:48,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:48,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:48,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:17:48,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:17:48,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:17:49,254 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:17:50,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:52,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:53,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:17:54,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:56,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:58,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:17:59,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:00,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:02,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:03,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:18:03,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:18:03,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:18:04,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:05,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:06,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:08,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:10,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:10,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:12,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:14,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:15,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:16,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:18,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:18,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:18:18,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:18:18,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:18:19,258 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:18:20,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:21,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:22,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:24,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:26,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:26,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:28,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:30,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:32,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:32,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:33,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:18:33,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:18:33,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:18:34,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:36,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:37,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:38,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:40,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:42,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:43,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:44,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:46,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:48,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:48,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:48,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:18:48,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:18:48,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:18:49,260 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:18:50,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:52,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:53,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:18:54,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:56,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:58,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:18:59,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:00,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:02,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:03,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:19:03,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:19:03,934 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:19:04,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:04,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:06,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:08,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:09,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:10,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:12,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:14,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:14,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:16,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:18,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:18,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:19:18,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:19:18,933 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:19:19,263 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:19:20,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:20,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:22,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:24,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:25,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:26,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:28,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:28,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:19:28,548 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-17 23:19:28,555 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-17 23:19:28,677 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-17 23:19:29,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-17 23:19:30,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:30,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:34,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:19:34,822 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:19:35,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:19:35,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:40,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:41,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:42,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:46,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:49,266 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:19:50,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:19:50,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:19:50,011 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:19:50,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:19:50,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:19:50,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-17 23:19:50,283 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-17 23:19:50,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:19:50,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_2_239e091867f620d0e20a.table.json 2025-09-17 23:19:50,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-17 23:19:50,858 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/4z5lqknr-media/table/rollouts_2_239e091867f620d0e20a.table.json 2025-09-17 23:19:50,989 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpyrgdepvc 2025-09-17 23:19:51,645 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTM0OTY2OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTIyMTM0NA==', 'versionIndex': 0}}} 2025-09-17 23:19:51,645 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-17 23:19:51,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:51,646 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_2_239e091867f620d0e20a.table.json with policy now 2025-09-17 23:19:51,946 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/877nvr8m-media/table/rollouts_2_239e091867f620d0e20a.table.json 2025-09-17 23:19:52,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:54,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:56,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:19:56,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:19:58,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:00,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:01,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:02,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:04,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:20:04,140 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:20:04,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:04,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:20:06,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:07,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:08,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:10,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:12,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:13,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:14,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:16,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:18,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:18,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:19,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:20:19,140 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:20:19,390 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:20:19,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:20:20,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:22,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:24,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:24,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:26,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:28,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:29,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:30,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:32,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:34,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:20:34,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:20:34,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:34,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:20:34,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:36,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:38,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:40,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:40,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:42,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:44,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:46,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:46,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:48,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:49,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:20:49,140 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:20:49,369 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:20:49,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:20:50,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:51,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:52,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:54,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:56,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:20:57,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:20:58,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:00,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:02,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:02,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:04,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:21:04,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:21:04,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:04,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:21:06,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:08,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:08,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:10,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:12,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:13,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:14,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:16,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:18,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:19,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:21:19,140 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:21:19,300 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:21:19,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:19,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:21:20,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:22,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:24,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:24,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:26,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:28,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:30,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:30,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:32,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:34,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:21:34,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:21:34,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:34,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:21:36,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:36,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:38,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:40,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:41,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:42,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:44,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:46,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:47,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:48,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:49,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:21:49,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:21:49,362 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:21:49,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:21:50,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:52,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:52,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:54,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:56,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:21:58,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:21:58,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:00,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:02,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:03,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:04,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:22:04,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:22:04,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:04,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:22:06,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:08,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:09,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:10,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:12,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:14,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:14,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:16,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:18,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:19,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:22:19,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:22:19,349 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:22:19,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:22:19,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:20,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:22,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:24,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:25,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:26,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:28,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:30,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:31,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:32,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:34,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:22:34,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:22:34,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:34,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:22:36,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:36,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:38,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:40,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:42,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:42,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:44,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:46,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:47,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:48,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:49,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:22:49,141 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:22:49,345 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:22:49,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:22:50,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:52,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:53,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:22:54,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:56,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:58,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:22:59,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:00,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:02,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:04,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:23:04,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:23:04,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:04,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:04,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:23:06,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:08,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:10,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:10,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:12,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:14,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:15,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:16,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:18,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:19,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:23:19,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:23:19,327 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:23:19,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:23:20,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:21,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:22,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:24,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:26,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:26,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:28,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:30,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:32,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:32,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:34,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:23:34,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:23:34,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:34,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:23:36,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:37,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:38,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:40,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:42,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:43,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:44,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:46,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:48,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:48,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:49,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:23:49,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:23:49,321 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:23:49,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:23:50,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:52,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:54,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:23:54,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:56,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:58,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:23:59,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:00,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:02,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:04,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:24:04,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:24:04,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:04,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:24:04,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:06,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:08,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:10,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:10,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:12,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:14,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:15,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:16,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:18,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:19,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:24:19,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:24:19,333 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:24:19,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:24:20,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:20,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:22,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:24,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:26,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:26,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:28,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:30,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:31,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:32,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:34,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:24:34,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:24:34,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:34,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:24:36,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:37,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:38,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:40,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:42,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:42,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:44,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:46,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:48,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:48,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:49,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:24:49,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:24:49,358 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:24:49,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:24:50,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:52,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:54,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:24:54,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:56,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:58,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:24:59,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:00,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:02,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:04,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:25:04,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:25:04,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:04,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:25:04,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:06,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:08,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:10,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:10,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:12,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:14,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:16,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:16,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:18,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:19,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:25:19,142 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:25:19,330 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:25:19,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:25:20,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:21,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:22,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:24,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:26,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:27,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:28,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:30,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:32,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:32,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:34,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:25:34,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:25:34,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:25:34,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:36,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:38,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:38,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:40,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:42,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:43,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:44,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:46,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:48,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:49,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:25:49,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:25:49,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:49,333 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:25:49,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:25:50,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:52,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:54,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:54,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:25:56,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:25:58,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:00,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:00,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:02,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:04,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:26:04,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:26:04,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:26:04,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:06,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:06,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:08,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:10,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:11,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:12,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:14,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:16,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:17,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:18,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:19,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:26:19,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:26:19,336 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:26:19,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:26:20,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:22,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:22,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:24,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:26,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:28,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:28,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:30,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:32,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:33,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:34,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:26:34,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:26:34,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:26:34,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:36,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:38,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:39,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:40,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:42,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:44,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:44,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:46,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:48,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:49,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:26:49,143 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:26:49,361 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:26:49,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:26:49,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:50,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:52,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:54,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:55,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:26:56,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:26:58,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:00,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:01,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:02,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:04,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:27:04,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:27:04,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:27:04,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:06,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:06,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:08,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:10,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:12,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:12,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:14,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:16,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:17,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:18,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:19,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:27:19,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:27:19,342 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:27:19,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:27:20,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:22,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:23,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:24,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:26,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:28,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:29,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:30,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:32,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:34,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:27:34,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:27:34,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:34,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:27:34,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:36,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:38,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:40,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:40,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:42,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:44,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:45,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:46,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:48,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:49,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:27:49,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:27:49,364 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:27:49,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:27:50,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:51,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:52,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:54,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:56,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:27:56,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:27:58,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:00,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:02,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:02,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:04,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:28:04,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:28:04,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:28:04,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:06,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:07,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:08,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:10,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:12,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:13,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:14,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:16,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:18,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:18,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:19,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:28:19,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:28:19,349 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:28:19,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:28:20,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:22,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:24,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:24,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:26,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:28,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:29,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:30,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:32,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:34,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:28:34,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:28:34,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:28:34,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:34,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:36,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:38,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:40,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:40,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:42,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:44,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:46,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:46,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:48,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:49,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:28:49,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:28:49,355 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:28:49,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:28:50,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:52,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:52,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:54,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:56,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:28:57,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:28:58,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:00,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:02,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:03,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:04,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:29:04,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:29:04,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:29:04,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:06,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:08,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:08,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:10,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:12,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:14,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:14,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:16,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:18,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:19,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:29:19,144 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:29:19,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:19,354 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:29:19,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:29:20,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:22,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:24,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:25,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:26,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:28,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:30,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:30,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:32,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:34,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:29:34,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:29:34,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:29:34,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:36,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:36,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:38,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:40,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:41,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:42,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:44,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:46,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:47,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:48,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:49,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:29:49,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:29:49,364 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:29:49,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:29:50,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:52,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:52,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:54,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:56,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:29:58,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:29:58,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:00,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:02,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:03,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:04,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:30:04,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:30:04,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:30:04,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:06,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:08,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:09,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:10,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:12,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:14,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:15,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:16,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:18,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:19,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:30:19,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:30:19,360 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:30:19,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:30:20,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:20,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:22,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:24,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:26,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:26,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:28,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:30,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:31,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:32,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:34,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:30:34,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:30:34,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:30:34,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:36,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:37,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:38,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:40,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:42,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:42,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:44,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:46,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:48,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:48,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:49,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:30:49,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:30:49,366 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:30:49,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:30:50,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:52,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:53,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:30:54,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:56,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:58,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:30:59,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:00,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:02,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:04,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:31:04,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:31:04,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:04,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:31:04,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:06,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:08,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:09,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:10,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:12,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:14,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:14,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:16,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:18,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:19,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:31:19,145 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:31:19,366 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:31:19,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:31:19,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:20,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:22,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:24,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:25,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:26,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:28,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:30,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:31,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:32,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:34,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:31:34,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:31:34,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:31:34,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:36,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:36,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:38,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:40,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:42,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:42,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:44,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:46,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:47,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:48,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:49,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:31:49,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:31:49,368 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:31:49,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:31:50,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:52,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:53,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:54,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:56,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:31:58,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:31:58,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:00,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:02,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:04,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:32:04,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:32:04,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:04,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:32:04,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:06,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:08,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:09,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:10,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:12,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:14,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:15,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:16,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:18,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:19,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:32:19,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:32:19,371 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:32:19,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:32:20,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:21,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:22,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:24,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:26,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:26,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:28,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:30,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:32,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:32,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:34,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:32:34,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:32:34,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:32:34,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:36,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:37,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:38,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:40,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:42,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:43,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:44,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:46,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:48,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:48,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:49,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:32:49,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:32:49,378 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:32:49,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:32:50,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:52,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:54,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:32:54,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:56,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:58,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:32:59,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:00,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:02,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:04,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:33:04,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:33:04,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:33:04,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:04,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:06,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:08,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:10,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:10,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:12,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:14,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:16,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:16,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:18,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:19,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:33:19,146 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:33:19,379 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:33:19,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:33:20,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:21,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:22,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:24,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:26,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:27,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:28,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:30,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:32,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:32,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:34,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:33:34,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:33:34,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:33:34,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:36,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:38,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:38,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:40,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:42,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:44,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:44,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:46,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:48,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:49,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:33:49,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:33:49,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:49,382 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:33:49,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:33:50,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:52,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:54,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:55,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:33:56,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:33:58,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:00,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:00,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:02,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:04,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:34:04,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:34:04,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:34:04,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:06,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:06,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:08,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:10,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:11,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:12,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:14,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:16,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:17,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:17,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:17,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:18,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:18,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:18,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:34:18,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:19,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:34:19,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:34:19,385 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:34:19,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:34:20,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:22,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:22,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:24,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:27,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:28,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:29,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:31,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:33,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:33,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:34,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:34:34,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:34:34,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:34:35,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:37,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:39,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:39,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:41,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:43,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:45,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:45,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:47,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:49,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:49,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:34:49,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:34:49,389 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:34:49,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:34:50,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:51,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:53,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:55,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:56,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:34:57,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:34:59,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:01,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:01,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:03,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:04,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:35:04,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:35:04,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:35:05,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:07,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:07,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:09,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:11,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:12,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:13,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:15,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:17,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:18,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:19,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:19,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:35:19,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:35:19,392 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:35:19,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:35:21,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:23,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:23,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:25,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:27,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:29,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:29,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:31,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:33,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:34,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:35:34,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:35:34,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:35:34,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:35,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:37,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:39,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:40,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:41,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:43,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:45,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:46,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:47,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:49,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:49,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:35:49,147 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:35:49,395 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:35:49,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:35:51,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:51,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:53,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:55,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:57,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:35:57,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:35:59,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:01,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:02,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:03,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:04,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:36:04,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:36:04,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:36:05,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:07,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:08,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:09,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:11,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:13,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:13,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:15,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:17,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:19,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:19,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:36:19,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:36:19,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:19,398 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:36:19,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:36:21,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:23,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:24,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:25,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:27,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:29,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:30,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:31,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:33,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:34,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:36:34,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:36:34,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:36:35,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:35,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:37,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:39,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:41,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:41,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:43,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:45,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:46,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:47,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:49,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:49,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:36:49,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:36:49,401 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:36:49,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:36:51,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:52,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:53,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:55,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:57,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:36:58,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:36:59,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:01,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:03,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:03,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:04,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:37:04,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:37:04,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:37:05,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:07,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:09,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:09,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:11,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:13,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:14,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:15,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:17,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:19,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:19,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:37:19,148 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:37:19,404 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:37:19,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:37:19,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:21,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:23,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:25,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:25,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:27,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:29,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:31,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:31,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:33,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:34,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:37:34,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:37:34,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:37:35,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:36,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:37,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:39,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:41,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:42,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:43,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:45,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:47,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:47,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:49,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:49,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:37:49,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:37:49,406 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:37:49,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:37:51,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:53,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:53,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:55,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:57,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:37:58,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:37:59,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:01,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:03,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:04,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:38:04,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:38:04,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:04,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:38:05,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:07,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:09,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:10,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:11,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:13,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:15,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:15,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:17,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:19,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:38:19,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:38:19,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:19,412 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:38:19,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:38:21,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:21,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:23,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:25,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:26,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:27,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:29,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:31,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:32,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:33,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:34,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:38:34,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:38:34,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:38:35,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:37,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:37,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:39,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:41,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:43,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:43,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:45,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:47,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:48,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:49,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:38:49,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:38:49,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:49,412 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:38:49,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:38:51,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:53,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:54,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:38:55,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:57,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:59,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:38:59,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:01,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:03,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:04,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:39:04,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:39:04,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:39:04,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:05,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:07,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:09,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:10,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:11,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:13,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:15,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:16,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:17,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:19,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:39:19,149 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:39:19,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:19,415 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:39:19,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:39:21,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:21,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:23,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:25,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:27,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:27,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:29,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:31,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:33,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:33,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:34,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:39:34,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:39:34,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:39:35,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:37,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:38,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:39,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:41,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:43,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:44,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:45,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:47,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:49,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:39:49,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:39:49,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:49,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:49,418 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:39:49,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:39:51,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:53,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:55,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:39:55,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:57,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:39:59,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:00,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:01,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:03,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:04,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:40:04,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:40:04,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:40:05,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:06,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:07,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:09,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:11,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:11,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:13,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:15,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:17,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:17,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:19,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:40:19,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:40:19,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:19,422 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:40:19,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:40:21,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:22,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:23,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:25,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:27,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:28,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:29,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:31,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:33,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:33,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:34,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:40:34,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:40:34,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:40:35,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:37,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:39,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:39,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:41,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:43,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:44,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:45,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:47,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:49,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:40:49,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:40:49,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:49,423 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:40:49,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:40:49,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:51,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:53,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:55,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:40:55,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:57,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:40:59,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:00,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:01,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:03,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:04,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:41:04,151 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:41:04,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:41:05,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:05,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:07,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:09,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:11,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:11,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:13,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:15,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:16,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:17,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:19,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:41:19,151 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:41:19,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:19,426 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:41:19,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:41:21,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:22,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:23,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:25,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:27,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:27,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:29,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:31,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:33,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:33,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:34,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:41:34,151 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:41:34,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:41:35,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:37,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:38,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:39,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:41,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:43,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:44,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:45,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:47,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:49,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:41:49,150 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:41:49,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:49,429 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:41:49,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:41:49,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:51,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:53,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:55,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:55,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:41:57,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:41:59,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:01,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:01,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:03,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:04,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:42:04,151 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:42:04,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:42:05,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:06,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:07,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:09,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:11,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:12,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:13,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:15,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:17,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:17,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:19,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:42:19,151 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:42:19,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:19,434 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:42:19,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:42:21,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:23,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:23,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:25,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:27,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:28,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:29,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:31,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:33,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:34,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:42:34,152 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:42:34,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:34,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:42:35,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:37,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:39,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:39,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:41,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:43,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:45,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:45,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:47,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:49,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:42:49,151 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:42:49,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:49,435 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:42:49,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:42:50,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:51,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:53,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:55,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:56,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:42:57,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:42:59,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:01,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:01,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:03,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:04,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:43:04,152 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:43:04,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:43:05,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:07,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:07,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:09,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:11,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:13,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:13,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:15,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:17,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:18,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:19,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:43:19,152 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:43:19,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:19,437 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:43:19,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:43:21,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:23,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:24,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:25,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:27,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:29,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:29,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:31,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:33,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:34,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:43:34,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:43:34,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:43:34,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:35,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:37,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:39,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:40,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:41,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:43,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:45,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:46,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:47,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:49,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:43:49,152 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:43:49,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:49,441 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:43:49,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:43:51,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:51,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:53,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:55,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:57,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:43:57,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:43:59,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:01,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:02,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:03,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:04,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:44:04,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:44:04,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:44:05,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:07,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:08,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:09,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:11,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:13,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:13,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:15,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:17,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:19,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:44:19,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:44:19,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:19,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:19,444 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:44:19,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:44:21,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:23,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:24,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:25,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:27,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:29,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:30,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:31,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:33,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:34,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:44:34,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:44:34,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:44:35,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:36,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:37,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:39,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:41,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:41,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:43,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:45,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:47,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:47,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:49,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:44:49,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:44:49,446 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:44:49,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:49,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:44:51,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:52,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:53,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:55,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:57,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:44:58,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:44:59,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:01,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:03,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:03,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:04,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:45:04,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:45:04,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:45:05,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:07,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:09,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:09,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:11,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:13,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:14,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:15,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:17,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:19,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:45:19,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:45:19,449 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:45:19,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:19,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:45:19,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:21,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:23,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:25,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:25,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:27,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:29,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:31,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:31,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:33,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:34,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:45:34,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:45:34,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:45:35,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:36,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:37,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:39,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:41,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:42,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:43,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:45,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:47,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:48,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:49,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:45:49,153 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:45:49,452 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:45:49,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:49,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:45:51,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:53,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:53,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:55,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:57,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:45:59,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:45:59,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:01,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:03,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:04,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:46:04,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:46:04,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:04,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:46:05,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:07,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:09,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:10,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:11,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:13,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:15,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:15,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:17,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:19,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:46:19,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:46:19,455 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:46:19,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:19,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:46:21,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:21,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:23,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:25,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:26,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:27,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:29,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:31,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:32,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:33,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:34,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:46:34,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:46:34,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:46:35,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:37,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:37,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:39,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:41,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:43,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:43,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:45,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:47,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:48,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:49,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:46:49,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:46:49,458 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:46:49,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:49,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:46:51,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:53,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:54,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:46:55,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:57,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:59,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:46:59,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:01,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:03,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:04,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:47:04,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:47:04,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:47:04,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:05,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:07,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:09,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:10,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:11,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:13,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:15,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:15,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:17,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:19,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:47:19,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:47:19,460 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:47:19,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:19,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:47:20,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:21,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:23,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:25,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:26,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:27,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:29,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:31,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:31,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:33,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:34,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:47:34,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:47:34,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:47:35,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:37,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:37,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:39,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:41,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:42,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:43,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:45,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:47,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:48,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:49,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:47:49,154 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:47:49,466 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:47:49,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:49,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:47:51,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:53,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:53,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:55,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:57,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:47:59,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:47:59,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:01,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:03,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:04,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:48:04,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:48:04,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:48:04,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:05,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:07,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:09,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:10,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:11,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:13,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:15,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:16,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:17,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:19,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:48:19,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:48:19,468 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:48:19,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:19,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:48:21,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:21,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:23,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:25,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:27,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:27,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:29,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:31,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:32,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:33,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:34,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:48:34,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:48:34,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:48:35,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:37,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:38,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:39,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:41,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:43,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:43,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:45,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:47,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:49,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:48:49,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:48:49,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:49,470 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:48:49,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:48:49,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:51,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:53,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:54,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:48:55,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:57,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:48:59,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:00,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:01,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:03,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:04,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:49:04,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:49:04,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:49:05,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:05,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:07,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:09,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:11,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:11,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:13,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:15,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:17,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:17,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:19,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:49:19,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:49:19,473 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:49:19,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:49:19,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:21,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:22,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:23,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:25,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:27,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:28,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:29,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:31,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:33,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:33,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:34,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:49:34,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:49:34,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:49:35,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:37,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:39,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:39,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:41,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:43,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:44,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:45,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:47,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:49,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:49:49,155 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:49:49,478 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:49:49,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:49:49,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:49,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:51,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:53,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:55,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:55,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:49:57,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:49:59,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:01,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:01,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:03,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:04,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:50:04,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:50:04,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:50:05,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:06,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:07,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:09,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:11,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:12,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:13,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:15,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:17,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:17,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:19,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:50:19,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:50:19,478 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:50:19,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:50:19,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:21,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:23,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:23,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:25,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:27,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:28,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:29,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:31,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:33,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:34,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:50:34,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:50:34,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:34,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:50:35,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:37,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:39,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:40,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:41,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:43,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:45,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:45,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:47,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:49,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:50:49,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:50:49,481 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:50:49,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:50:49,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:51,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:51,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:53,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:55,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:56,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:50:57,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:50:59,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:01,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:02,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:03,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:04,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:51:04,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:51:04,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:51:05,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:07,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:07,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:09,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:11,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:13,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:13,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:15,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:17,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:18,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:19,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:51:19,156 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:51:19,484 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:51:19,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:51:19,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:21,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:23,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:24,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:25,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:27,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:29,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:29,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:31,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:33,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:34,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:51:34,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:51:34,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:51:34,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:35,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:37,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:39,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:40,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:41,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:43,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:45,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:46,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:47,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:49,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:51:49,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:51:49,486 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:51:49,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:51:49,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:51,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:52,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:53,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:55,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:57,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:51:57,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:51:59,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:01,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:03,054 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:03,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:04,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:52:04,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:52:04,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:52:05,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:07,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:08,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:09,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:11,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:13,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:14,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:15,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:17,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:19,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:52:19,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:52:19,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:19,489 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:52:19,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:52:19,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:21,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:23,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:25,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:25,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:27,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:29,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:30,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:31,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:33,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:34,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:52:34,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:52:34,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:52:35,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:36,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:37,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:39,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:41,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:41,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:43,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:45,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:47,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:47,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:49,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:52:49,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:52:49,492 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:52:49,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:52:49,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:51,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:52,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:53,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:55,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:57,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:52:58,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:52:59,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:01,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:03,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:03,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:04,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:53:04,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:53:04,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:53:05,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:07,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:09,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:09,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:11,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:13,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:14,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:15,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:17,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:19,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:53:19,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:53:19,495 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:53:19,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:53:19,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:19,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:21,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:23,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:25,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:25,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:27,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:29,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:30,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:31,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:33,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:34,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:53:34,158 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:53:34,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:53:35,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:35,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:37,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:39,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:41,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:41,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:43,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:45,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:46,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:47,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:49,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:53:49,157 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:53:49,498 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:53:49,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:53:49,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:51,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:52,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:53,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:55,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:57,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:53:57,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:53:59,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:01,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:03,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:03,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:04,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:54:04,158 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:54:04,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:54:05,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:07,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:08,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:09,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:11,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:13,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:14,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:15,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:17,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:19,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:54:19,158 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:54:19,500 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:54:19,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:54:19,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:19,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:21,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:23,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:25,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:25,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:27,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:29,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:31,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:31,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:33,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:34,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:54:34,158 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:54:34,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:54:35,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:36,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:37,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:39,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:41,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:42,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:43,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:45,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:47,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:47,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:49,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:54:49,158 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:54:49,504 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:54:49,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:54:49,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:52,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:53,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:54,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:56,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:54:58,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:54:58,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:00,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:02,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:04,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:55:04,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:55:04,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:04,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:55:04,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:06,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:08,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:09,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:10,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:12,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:14,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:15,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:16,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:18,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:19,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:55:19,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:55:19,508 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:55:19,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:55:20,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:20,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:22,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:24,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:26,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:26,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:28,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:30,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:32,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:32,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:34,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:55:34,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:55:34,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:55:34,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:36,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:37,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:38,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:40,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:42,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:43,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:44,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:46,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:48,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:48,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:49,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:55:49,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:55:49,509 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:55:49,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:55:50,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:52,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:54,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:55:54,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:56,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:58,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:55:59,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:00,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:02,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:04,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:56:04,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:56:04,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:56:04,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:04,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:06,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:08,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:10,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:10,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:12,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:14,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:16,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:16,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:18,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:19,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:56:19,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:56:19,512 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:56:19,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:56:20,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:21,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:22,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:24,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:26,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:27,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:28,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:30,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:32,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:32,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:34,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:56:34,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:56:34,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:56:34,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:36,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:38,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:38,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:40,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:42,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:43,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:44,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:46,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:48,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:49,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:56:49,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:56:49,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:49,514 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:56:49,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:56:50,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:52,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:54,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:55,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:56:56,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:56:58,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:00,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:00,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:02,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:04,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:57:04,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:57:04,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:57:04,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:06,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:07,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:09,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:11,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:11,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:13,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:15,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:17,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:17,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:19,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:19,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:57:19,159 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:57:19,520 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:57:19,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:57:21,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:21,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:21,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:57:22,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:23,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:25,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:27,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:27,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:29,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:31,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:32,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:33,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:34,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:57:34,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:57:34,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:57:35,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:37,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:38,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:39,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:41,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:43,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:43,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:45,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:47,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:49,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:49,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:57:49,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:57:49,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:49,521 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:57:49,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:57:51,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:53,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:54,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:57:55,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:57,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:57:59,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:00,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:01,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:03,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:04,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:58:04,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:58:04,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:58:05,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:05,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:07,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:09,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:11,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:11,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:13,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:15,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:16,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:17,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:19,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:19,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:58:19,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:58:19,524 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:58:19,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:58:21,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:22,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:23,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:25,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:27,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:27,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:29,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:31,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:33,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:33,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:34,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:58:34,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:58:34,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:58:35,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:37,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:38,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:39,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:41,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:43,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:44,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:45,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:47,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:49,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:49,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:58:49,160 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:58:49,527 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:58:49,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:49,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:58:51,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:53,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:54,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:58:55,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:57,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:59,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:58:59,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:01,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:03,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:04,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:59:04,161 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:59:04,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:59:04,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:05,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:07,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:09,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:10,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:11,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:13,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:15,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:15,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:17,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:19,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:19,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:59:19,161 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:59:19,530 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:59:19,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:59:20,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:21,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:23,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:25,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:25,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:27,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:27,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:59:27,617 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-17 23:59:27,624 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-17 23:59:27,633 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-17 23:59:28,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-17 23:59:29,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:31,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:34,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:59:34,542 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:59:35,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:59:37,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:41,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:43,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:48,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:49,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:59:49,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:59:49,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-17 23:59:49,468 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-17 23:59:49,616 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-17 23:59:49,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-17 23:59:51,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:51,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-17 23:59:51,268 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-17 23:59:51,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-17 23:59:52,094 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpw1c6xnia 2025-09-17 23:59:52,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_3_71cc41780ac1c30b1113.table.json 2025-09-17 23:59:52,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-17 23:59:52,905 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTQ3OTIyOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTM0OTY2OQ==', 'versionIndex': 1}}} 2025-09-17 23:59:52,906 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-17 23:59:52,906 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_3_71cc41780ac1c30b1113.table.json with policy now 2025-09-17 23:59:53,146 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/81jaa8o5-media/table/rollouts_3_71cc41780ac1c30b1113.table.json 2025-09-17 23:59:53,461 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/s3thlw8i-media/table/rollouts_3_71cc41780ac1c30b1113.table.json 2025-09-17 23:59:53,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:55,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:57,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-17 23:59:58,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-17 23:59:59,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:01,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:03,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:04,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:04,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:00:04,468 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:00:04,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:00:05,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:07,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:09,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:09,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:11,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:13,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:15,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:15,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:17,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:19,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:19,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:00:19,468 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:00:19,589 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:00:19,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:00:21,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:21,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:23,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:25,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:26,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:27,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:29,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:31,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:32,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:33,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:34,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:00:34,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:00:34,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:00:35,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:37,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:37,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:39,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:41,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:43,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:43,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:45,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:47,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:48,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:49,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:49,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:00:49,468 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:00:49,619 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:00:49,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:00:51,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:53,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:54,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:00:55,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:57,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:59,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:00:59,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:01,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:03,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:04,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:01:04,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:01:04,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:01:05,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:05,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:07,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:09,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:10,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:11,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:13,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:15,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:16,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:17,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:19,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:19,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:01:19,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:01:19,644 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:01:19,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:01:21,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:21,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:23,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:25,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:27,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:27,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:29,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:31,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:32,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:33,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:34,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:01:34,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:01:34,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:01:35,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:37,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:38,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:39,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:41,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:43,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:43,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:45,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:47,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:49,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:49,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:01:49,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:01:49,640 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:01:49,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:49,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:01:51,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:53,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:55,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:01:55,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:57,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:01:59,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:00,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:01,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:03,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:04,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:02:04,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:02:04,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:02:05,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:06,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:07,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:09,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:11,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:11,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:13,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:15,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:17,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:17,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:19,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:19,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:02:19,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:02:19,605 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:02:19,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:02:21,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:22,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:23,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:25,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:27,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:28,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:29,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:31,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:33,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:33,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:34,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:02:34,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:02:34,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:02:35,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:37,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:39,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:39,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:41,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:43,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:44,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:45,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:47,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:49,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:49,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:02:49,469 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:02:49,643 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:02:49,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:02:50,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:51,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:53,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:55,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:55,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:02:57,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:02:59,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:01,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:01,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:03,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:04,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:03:04,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:03:04,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:03:05,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:07,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:07,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:09,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:11,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:12,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:13,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:15,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:17,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:18,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:19,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:19,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:03:19,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:03:19,649 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:03:19,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:03:21,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:23,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:23,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:25,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:27,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:29,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:29,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:31,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:33,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:34,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:03:34,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:03:34,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:34,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:03:35,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:37,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:39,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:40,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:41,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:43,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:45,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:45,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:47,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:49,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:49,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:03:49,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:03:49,590 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:03:49,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:03:51,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:51,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:53,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:55,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:56,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:03:57,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:03:59,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:01,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:02,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:03,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:04,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:04:04,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:04:04,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:04:05,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:07,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:07,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:09,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:11,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:13,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:13,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:15,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:17,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:19,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:19,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:19,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:04:19,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:04:19,640 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:04:19,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:04:21,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:23,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:24,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:25,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:27,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:29,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:30,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:31,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:33,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:34,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:04:34,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:04:34,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:04:35,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:35,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:37,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:39,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:41,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:41,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:43,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:45,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:46,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:47,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:49,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:49,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:04:49,470 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:04:49,685 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:04:49,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:04:51,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:52,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:53,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:55,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:57,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:04:57,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:04:59,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:01,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:03,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:03,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:04,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:05:04,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:05:04,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:05:05,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:07,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:08,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:09,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:11,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:13,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:14,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:15,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:17,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:19,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:19,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:05:19,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:05:19,611 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:05:19,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:19,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:05:21,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:23,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:25,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:25,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:27,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:29,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:30,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:31,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:33,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:34,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:05:34,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:05:34,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:05:35,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:36,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:37,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:39,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:41,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:41,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:43,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:45,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:47,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:47,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:49,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:49,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:05:49,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:05:49,678 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:05:49,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:05:51,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:53,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:53,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:55,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:57,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:05:58,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:05:59,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:01,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:03,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:04,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:04,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:06:04,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:06:04,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:06:05,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:07,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:09,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:09,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:11,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:13,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:15,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:15,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:17,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:19,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:19,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:06:19,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:06:19,591 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:06:19,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:06:20,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:21,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:23,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:25,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:26,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:27,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:29,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:31,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:31,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:33,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:34,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:06:34,472 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:06:34,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:06:35,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:37,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:37,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:39,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:41,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:42,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:43,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:45,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:47,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:48,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:49,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:49,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:06:49,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:06:49,606 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:06:49,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:06:51,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:53,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:53,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:06:55,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:57,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:59,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:06:59,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:01,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:03,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:04,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:07:04,472 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:07:04,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:04,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:07:05,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:07,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:09,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:10,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:11,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:13,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:15,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:16,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:17,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:19,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:19,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:07:19,471 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:07:19,614 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:07:19,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:07:21,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:21,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:23,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:25,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:27,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:27,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:29,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:31,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:32,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:33,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:34,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:07:34,472 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:07:34,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:07:35,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:37,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:38,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:39,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:41,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:43,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:43,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:45,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:47,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:49,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:49,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:07:49,472 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:07:49,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:49,618 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:07:49,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:07:51,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:53,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:54,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:07:55,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:57,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:07:59,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:00,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:01,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:03,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:04,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:08:04,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:08:04,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:08:05,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:05,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:07,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:09,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:10,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:11,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:13,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:15,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:15,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:17,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:19,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:08:19,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:08:19,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:19,632 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:08:19,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:08:21,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:21,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:23,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:25,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:26,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:27,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:29,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:31,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:32,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:33,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:34,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:08:34,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:08:34,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:08:35,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:37,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:37,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:39,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:41,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:43,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:43,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:45,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:47,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:48,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:49,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:08:49,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:08:49,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:49,625 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:08:49,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:08:51,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:53,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:54,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:08:55,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:57,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:59,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:08:59,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:01,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:03,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:04,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:09:04,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:09:04,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:09:05,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:05,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:07,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:09,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:10,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:11,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:13,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:15,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:16,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:17,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:19,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:09:19,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:09:19,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:19,687 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:09:19,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:09:21,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:21,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:23,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:25,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:27,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:27,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:29,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:31,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:33,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:33,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:34,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:09:34,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:09:34,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:09:35,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:37,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:38,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:39,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:41,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:43,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:44,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:45,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:47,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:49,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:09:49,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:09:49,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:49,599 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:09:49,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:49,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:09:51,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:53,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:55,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:09:55,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:57,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:09:59,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:00,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:01,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:03,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:04,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:10:04,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:10:04,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:10:05,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:06,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:07,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:09,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:11,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:11,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:13,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:15,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:17,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:17,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:19,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:10:19,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:10:19,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:19,670 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:10:19,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:10:21,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:22,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:23,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:25,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:27,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:28,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:29,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:31,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:33,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:34,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:34,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:10:34,473 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:10:34,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:10:35,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:37,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:39,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:39,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:41,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:43,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:45,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:45,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:47,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:49,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:10:49,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:10:49,604 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:10:49,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:49,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:10:50,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:51,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:53,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:55,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:56,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:10:57,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:10:59,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:01,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:01,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:03,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:04,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:11:04,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:11:04,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:11:05,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:07,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:07,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:09,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:11,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:12,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:13,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:15,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:17,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:18,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:19,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:11:19,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:11:19,637 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:11:19,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:19,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:11:21,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:23,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:23,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:25,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:27,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:29,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:29,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:31,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:33,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:34,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:11:34,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:11:34,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:34,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:11:35,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:37,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:39,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:40,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:41,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:43,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:45,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:45,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:47,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:49,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:11:49,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:11:49,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:49,714 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:11:49,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:11:51,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:51,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:53,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:55,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:57,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:11:57,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:11:59,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:01,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:02,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:03,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:04,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:12:04,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:12:04,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:12:05,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:07,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:08,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:09,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:11,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:13,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:13,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:15,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:17,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:19,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:19,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:12:19,474 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:12:19,612 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:12:19,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:19,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:12:21,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:23,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:24,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:25,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:27,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:29,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:30,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:31,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:33,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:34,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:12:34,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:12:34,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:12:35,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:35,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:37,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:39,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:40,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:41,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:43,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:45,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:45,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:47,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:49,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:12:49,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:12:49,650 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:12:49,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:49,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:12:51,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:51,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:53,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:55,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:56,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:12:57,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:12:59,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:01,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:02,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:03,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:04,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:13:04,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:13:04,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:13:05,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:07,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:07,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:09,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:11,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:13,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:13,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:15,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:17,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:18,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:19,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:13:19,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:13:19,619 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:13:19,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:19,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:13:21,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:23,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:24,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:25,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:27,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:29,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:29,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:31,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:33,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:34,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:13:34,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:13:34,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:13:35,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:35,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:37,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:39,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:40,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:41,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:43,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:45,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:46,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:47,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:49,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:13:49,475 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:13:49,621 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:13:49,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:49,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:13:51,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:51,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:53,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:55,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:57,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:13:57,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:13:59,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:01,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:03,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:03,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:04,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:14:04,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:14:04,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:14:05,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:07,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:08,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:09,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:11,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:13,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:14,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:15,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:17,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:19,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:14:19,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:14:19,664 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:14:19,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:19,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:19,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:14:21,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:23,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:25,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:25,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:27,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:29,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:30,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:31,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:33,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:34,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:14:34,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:14:34,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:14:35,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:36,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:37,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:39,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:41,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:41,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:43,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:45,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:47,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:47,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:49,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:14:49,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:14:49,653 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:14:49,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:49,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:14:51,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:52,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:53,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:55,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:57,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:14:58,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:14:59,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:01,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:03,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:03,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:04,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:15:04,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:15:04,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:15:05,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:07,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:09,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:09,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:11,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:13,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:15,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:15,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:17,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:19,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:15:19,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:15:19,718 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:15:19,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:19,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:15:20,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:21,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:23,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:25,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:26,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:27,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:29,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:31,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:31,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:33,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:34,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:15:34,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:15:34,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:15:35,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:37,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:37,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:39,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:41,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:42,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:43,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:45,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:47,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:48,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:49,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:15:49,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:15:49,680 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:15:49,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:49,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:15:51,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:53,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:53,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:55,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:57,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:15:59,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:15:59,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:01,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:03,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:04,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:16:04,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:16:04,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:04,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:16:05,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:07,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:09,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:10,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:11,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:13,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:15,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:15,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:17,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:19,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:16:19,476 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:16:19,635 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:16:19,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:19,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:16:21,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:21,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:23,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:25,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:27,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:27,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:29,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:31,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:32,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:33,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:34,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:16:34,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:16:34,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:16:35,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:37,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:38,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:39,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:41,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:43,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:43,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:45,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:47,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:49,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:49,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:16:49,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:16:49,638 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:16:49,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:49,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:16:51,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:53,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:54,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:16:55,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:57,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:16:59,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:00,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:01,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:03,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:04,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:17:04,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:17:04,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:17:05,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:05,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:07,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:09,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:11,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:11,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:13,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:15,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:16,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:17,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:19,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:17:19,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:17:19,641 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:17:19,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:17:19,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:21,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:22,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:23,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:25,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:27,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:27,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:29,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:31,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:33,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:33,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:34,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:17:34,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:17:34,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:17:35,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:37,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:38,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:39,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:41,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:43,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:44,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:45,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:47,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:49,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:17:49,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:17:49,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:49,644 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:17:49,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:17:49,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:52,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:54,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:55,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:17:56,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:17:58,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:00,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:01,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:02,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:04,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:04,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:18:04,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:18:04,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:18:06,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:06,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:08,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:10,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:12,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:12,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:14,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:16,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:17,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:18,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:19,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:18:19,477 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:18:19,702 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:18:19,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:18:20,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:22,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:23,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:24,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:26,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:28,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:28,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:30,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:32,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:34,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:34,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:34,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:18:34,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:18:34,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:18:36,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:38,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:39,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:40,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:42,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:44,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:45,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:46,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:48,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:49,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:18:49,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:18:49,651 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:18:49,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:18:50,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:51,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:52,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:54,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:56,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:18:56,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:18:58,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:00,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:01,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:02,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:04,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:04,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:19:04,479 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:19:04,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:19:07,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:08,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:10,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:12,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:12,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:14,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:16,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:17,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:18,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:19,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:19:19,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:19:19,652 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:19:19,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:19:20,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:22,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:23,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:24,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:26,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:28,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:28,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:30,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:32,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:34,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:34,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:34,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:19:34,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:19:34,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:19:36,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:38,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:39,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:40,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:42,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:44,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:45,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:46,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:48,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:49,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:19:49,478 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:19:49,655 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:19:49,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:19:50,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:51,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:52,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:54,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:56,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:19:56,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:19:58,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:00,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:02,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:02,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:04,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:04,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:20:04,479 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:20:04,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:20:06,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:07,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:08,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:10,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:12,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:13,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:14,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:16,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:18,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:18,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:19,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:20:19,479 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:20:19,657 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:20:19,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:20:20,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:22,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:24,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:24,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:26,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:28,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:29,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:30,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:32,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:34,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:34,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:20:34,479 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:20:34,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:20:35,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:36,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:38,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:40,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:40,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:42,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:44,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:46,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:46,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:48,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:49,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:20:49,479 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:20:49,662 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:20:49,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:20:50,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:51,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:52,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:54,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:56,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:20:57,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:20:58,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:00,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:02,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:02,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:04,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:04,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:21:04,480 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:21:04,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:21:06,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:08,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:08,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:10,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:12,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:13,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:14,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:16,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:18,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:19,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:21:19,480 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:21:19,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:19,663 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:21:19,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:21:20,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:22,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:24,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:25,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:26,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:28,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:30,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:30,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:32,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:33,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:33,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:34,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:34,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:34,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:34,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:34,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:21:34,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:21:34,480 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:21:34,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:21:36,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:36,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:38,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:40,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:41,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:42,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:44,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:46,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:47,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:48,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:49,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:21:49,480 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:21:49,689 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:21:49,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:21:50,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:52,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:52,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:21:54,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:56,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:58,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:21:58,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:00,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:02,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:03,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:04,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:04,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:22:04,480 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:22:04,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:22:06,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:08,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:09,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:10,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:12,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:14,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:15,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:16,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:18,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:19,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:22:19,480 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:22:19,670 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:22:19,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:22:20,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:20,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:22,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:24,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:26,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:26,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:28,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:30,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:31,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:32,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:34,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:34,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:22:34,481 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:22:34,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:22:36,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:37,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:38,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:40,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:42,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:42,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:44,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:46,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:48,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:48,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:49,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:22:49,481 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:22:49,675 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:22:49,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:22:50,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:52,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:53,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:22:54,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:56,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:58,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:22:59,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:00,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:02,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:04,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:04,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:23:04,481 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:23:04,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:04,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:23:06,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:08,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:10,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:10,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:12,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:14,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:15,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:16,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:18,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:19,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:23:19,481 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:23:19,676 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:23:19,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:23:20,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:21,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:22,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:24,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:26,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:27,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:28,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:30,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:32,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:32,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:34,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:34,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:23:34,481 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:23:34,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:23:36,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:38,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:38,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:40,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:42,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:43,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:44,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:46,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:48,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:49,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:49,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:23:49,481 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:23:49,680 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:23:49,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:23:50,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:52,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:54,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:54,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:23:56,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:23:58,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:00,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:00,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:02,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:04,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:04,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:24:04,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:24:04,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:24:05,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:06,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:08,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:10,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:11,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:12,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:14,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:16,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:16,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:18,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:19,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:24:19,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:24:19,683 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:24:19,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:24:20,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:22,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:22,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:24,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:26,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:27,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:28,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:30,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:32,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:33,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:34,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:34,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:24:34,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:24:34,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:24:36,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:38,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:39,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:40,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:42,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:44,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:44,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:46,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:48,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:49,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:24:49,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:24:49,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:49,686 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:24:49,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:24:50,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:52,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:54,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:54,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:24:56,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:58,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:24:59,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:00,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:02,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:04,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:04,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:25:04,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:25:04,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:25:05,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:06,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:08,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:10,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:10,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:12,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:14,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:16,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:16,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:18,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:19,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:25:19,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:25:19,690 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:25:19,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:25:20,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:21,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:22,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:24,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:26,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:27,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:28,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:30,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:32,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:33,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:34,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:34,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:25:34,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:25:34,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:25:36,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:38,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:38,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:40,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:42,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:44,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:44,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:46,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:48,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:49,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:25:49,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:25:49,696 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:25:49,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:49,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:25:50,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:52,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:54,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:55,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:25:56,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:25:58,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:00,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:00,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:02,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:04,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:26:04,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:26:04,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:04,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:26:06,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:06,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:08,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:10,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:11,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:12,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:14,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:16,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:17,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:18,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:19,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:26:19,482 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:26:19,700 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:26:19,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:26:20,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:22,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:22,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:24,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:26,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:28,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:28,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:30,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:32,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:33,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:34,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:26:34,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:26:34,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:34,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:26:36,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:38,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:39,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:40,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:42,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:44,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:45,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:46,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:48,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:49,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:26:49,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:26:49,701 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:26:49,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:26:50,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:50,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:52,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:54,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:56,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:26:56,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:26:58,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:00,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:01,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:02,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:04,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:27:04,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:27:04,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:04,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:27:06,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:07,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:08,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:10,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:12,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:12,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:14,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:16,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:18,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:18,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:19,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:27:19,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:27:19,703 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:27:19,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:27:20,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:22,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:23,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:24,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:26,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:28,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:29,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:30,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:32,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:34,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:27:34,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:27:34,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:34,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:34,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:27:36,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:38,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:40,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:40,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:42,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:44,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:45,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:46,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:48,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:49,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:27:49,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:27:49,706 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:27:49,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:27:50,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:51,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:52,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:54,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:56,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:27:56,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:27:58,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:00,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:02,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:02,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:04,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:28:04,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:28:04,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:04,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:28:06,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:08,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:08,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:10,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:12,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:13,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:14,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:16,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:18,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:19,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:19,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:28:19,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:28:19,710 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:28:19,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:28:20,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:22,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:24,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:24,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:26,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:28,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:30,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:30,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:32,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:34,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:28:34,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:28:34,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:34,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:28:35,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:36,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:38,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:40,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:42,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:44,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:46,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:46,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:48,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:49,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:28:49,483 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:28:49,711 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:28:49,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:28:50,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:52,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:52,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:54,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:56,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:28:57,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:28:58,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:00,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:02,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:03,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:04,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:29:04,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:29:04,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:04,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:29:06,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:08,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:08,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:10,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:12,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:14,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:14,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:16,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:18,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:19,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:29:19,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:29:19,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:19,714 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:29:19,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:29:20,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:22,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:24,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:25,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:26,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:28,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:30,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:31,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:32,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:34,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:29:34,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:29:34,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:34,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:29:36,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:36,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:38,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:40,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:42,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:42,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:44,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:46,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:47,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:48,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:49,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:29:49,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:29:49,717 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:29:49,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:29:50,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:52,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:53,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:54,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:56,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:29:58,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:29:58,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:00,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:02,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:04,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:04,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:30:04,484 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:30:04,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:04,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:30:06,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:08,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:09,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:10,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:12,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:14,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:15,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:16,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:18,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:19,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:30:19,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:30:19,722 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:30:19,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:30:20,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:20,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:22,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:24,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:25,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:26,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:28,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:30,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:30,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:32,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:34,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:30:34,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:30:34,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:34,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:30:36,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:36,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:38,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:40,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:41,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:42,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:44,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:46,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:47,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:48,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:49,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:30:49,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:30:49,722 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:30:49,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:30:50,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:52,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:52,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:54,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:56,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:30:58,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:30:58,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:00,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:02,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:03,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:04,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:31:04,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:31:04,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:04,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:31:06,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:08,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:09,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:10,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:12,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:14,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:14,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:16,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:18,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:19,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:31:19,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:31:19,725 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:31:19,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:31:20,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:20,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:22,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:24,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:25,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:26,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:28,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:30,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:31,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:32,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:34,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:31:34,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:31:34,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:34,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:31:36,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:36,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:38,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:40,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:42,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:42,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:44,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:46,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:48,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:48,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:49,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:31:49,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:31:49,728 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:31:49,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:31:50,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:52,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:53,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:31:54,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:56,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:58,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:31:59,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:00,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:02,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:04,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:32:04,486 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:32:04,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:04,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:04,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:32:06,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:08,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:10,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:10,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:12,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:14,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:15,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:16,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:18,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:19,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:32:19,485 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:32:19,730 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:32:19,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:32:20,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:21,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:22,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:24,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:26,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:26,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:28,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:30,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:32,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:32,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:34,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:32:34,486 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:32:34,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:34,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:32:36,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:37,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:38,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:40,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:42,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:43,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:44,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:46,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:48,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:48,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:49,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:32:49,486 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:32:49,733 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:32:49,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:32:52,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:53,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:32:54,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:56,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:58,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:32:59,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:00,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:02,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:04,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:33:04,486 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:33:04,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:04,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:04,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:33:06,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:08,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:10,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:10,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:12,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:14,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:15,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:16,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:18,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:19,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:33:19,486 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:33:19,736 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:33:19,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:33:20,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:21,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:22,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:24,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:26,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:26,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:28,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:30,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:32,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:32,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:34,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:33:34,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:33:34,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:34,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:33:36,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:37,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:38,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:40,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:42,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:43,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:44,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:46,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:48,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:48,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:49,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:33:49,486 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:33:49,739 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:33:49,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:33:50,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:52,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:54,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:33:54,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:56,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:58,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:33:59,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:00,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:02,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:04,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:34:04,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:34:04,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:34:04,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:05,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:06,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:08,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:10,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:11,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:12,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:14,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:16,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:16,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:19,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:19,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:34:19,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:34:19,744 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:34:19,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:34:21,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:22,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:23,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:25,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:27,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:27,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:29,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:31,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:33,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:33,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:34,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:34:34,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:34:34,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:34:35,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:37,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:38,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:39,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:41,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:43,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:44,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:45,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:47,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:49,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:49,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:34:49,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:34:49,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:49,746 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:34:49,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:34:51,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:53,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:55,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:55,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:34:57,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:34:59,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:00,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:01,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:03,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:04,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:35:04,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:35:04,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:35:05,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:06,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:07,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:09,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:11,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:11,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:13,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:15,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:17,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:17,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:19,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:19,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:35:19,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:35:19,749 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:35:19,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:35:21,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:22,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:23,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:25,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:27,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:28,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:29,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:31,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:33,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:34,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:34,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:35:34,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:35:34,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:35:35,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:37,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:39,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:39,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:41,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:43,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:45,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:45,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:47,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:49,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:49,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:35:49,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:35:49,754 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:35:49,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:35:50,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:51,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:53,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:55,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:56,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:35:57,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:35:59,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:01,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:01,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:03,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:04,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:36:04,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:36:04,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:36:05,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:07,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:07,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:09,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:11,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:12,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:13,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:15,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:17,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:18,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:19,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:19,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:36:19,487 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:36:19,754 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:36:19,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:36:21,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:23,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:23,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:25,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:27,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:29,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:29,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:31,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:33,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:34,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:36:34,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:36:34,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:34,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:36:35,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:37,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:39,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:40,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:41,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:43,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:45,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:46,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:47,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:49,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:49,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:36:49,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:36:49,757 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:36:49,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:36:51,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:51,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:53,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:55,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:57,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:36:57,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:36:59,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:01,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:02,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:03,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:04,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:37:04,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:37:04,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:37:05,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:07,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:08,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:09,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:11,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:13,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:13,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:15,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:17,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:19,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:19,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:19,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:37:19,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:37:19,761 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:37:19,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:37:21,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:23,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:24,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:25,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:27,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:29,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:30,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:31,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:33,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:34,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:37:34,489 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:37:34,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:37:35,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:36,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:37,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:39,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:41,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:41,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:43,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:45,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:47,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:47,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:49,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:49,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:37:49,488 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:37:49,765 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:37:49,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:37:51,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:52,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:53,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:55,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:57,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:37:58,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:37:59,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:01,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:03,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:03,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:04,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:38:04,489 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:38:04,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:38:05,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:07,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:09,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:09,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:11,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:13,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:14,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:15,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:17,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:19,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:19,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:38:19,489 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:38:19,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:19,766 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:38:19,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:38:21,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:23,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:24,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:25,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:27,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:29,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:29,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:31,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:33,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:34,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:38:34,489 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:38:34,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:38:35,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:35,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:37,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:39,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:40,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:41,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:43,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:45,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:46,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:47,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:49,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:49,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:38:49,489 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:38:49,768 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:38:49,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:38:51,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:52,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:53,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:55,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:57,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:38:57,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:38:59,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:01,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:02,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:03,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:04,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:39:04,490 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:39:04,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:39:05,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:07,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:07,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:09,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:11,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:13,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:13,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:15,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:17,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:18,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:19,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:19,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:39:19,489 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:39:19,773 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:39:19,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:39:21,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:23,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:23,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:25,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:27,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:29,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:29,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:31,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:33,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:34,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:39:34,490 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:39:34,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:34,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:39:35,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:39:35,129 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 00:39:35,136 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 00:39:35,146 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 00:39:35,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 00:39:35,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:37,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:40,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:45,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:47,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:39:49,775 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:39:50,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:39:50,023 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:39:50,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:50,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:39:55,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:39:57,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:39:57,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:39:57,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 00:39:57,932 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 00:39:57,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:39:58,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_4_1db01a832c96ad087afc.table.json 2025-09-18 00:39:58,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 00:39:59,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:40:04,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: stop_status 2025-09-18 00:40:04,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:40:11,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:40:16,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:40:19,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:40:25,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:40:30,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:40:34,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:40:40,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:40:45,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:40:49,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:40:55,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:00,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:04,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:41:10,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:15,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:19,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:41:25,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:30,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:34,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:41:40,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:45,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:41:49,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:41:55,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:00,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:04,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:42:09,747 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmprj5d9mu4 2025-09-18 00:42:10,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:15,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:19,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:42:25,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:30,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:34,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:42:40,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:45,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:42:49,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:42:55,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:00,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:04,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:43:10,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:15,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:19,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:43:25,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:30,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:34,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:43:40,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:45,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:43:49,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:43:55,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:00,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:04,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:44:10,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:15,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:19,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:44:20,847 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/pzl8gn3o-media/table/rollouts_4_1db01a832c96ad087afc.table.json 2025-09-18 00:44:25,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:30,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:34,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:44:40,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:45,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:44:49,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:44:55,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:00,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:04,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:45:10,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:15,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:19,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:45:25,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:30,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:34,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:45:40,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:45,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:45:47,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:48,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 00:45:49,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:45:55,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:00,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:04,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:46:10,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:15,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:19,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:46:25,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:30,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:32,366 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTYwMDA2MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTQ3OTIyOA==', 'versionIndex': 2}}} 2025-09-18 00:46:32,366 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 00:46:32,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:46:32,367 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_4_1db01a832c96ad087afc.table.json with policy now 2025-09-18 00:46:32,367 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: stop_status 2025-09-18 00:46:32,621 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,623 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,626 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,628 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,629 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,630 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,632 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,633 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,635 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,636 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,637 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,639 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,640 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:32,675 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/3ouisoy4-media/table/rollouts_4_1db01a832c96ad087afc.table.json 2025-09-18 00:46:33,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:34,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:46:35,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:37,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:46:37,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:39,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:40,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:41,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:42,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:46:43,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:45,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:45,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:47,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:48,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:46:49,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:49,816 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:46:49,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:46:51,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:53,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:54,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:46:55,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:46:55,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:57,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:46:59,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:46:59,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:00,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:01,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:03,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:04,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:47:05,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:05,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:07,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:09,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:10,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:10,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:11,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:13,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:15,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:15,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:16,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:17,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:19,820 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:47:19,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:19,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:47:21,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:21,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:23,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:25,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:25,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:27,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:27,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:29,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:30,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:31,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:32,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:33,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:34,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:47:35,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:37,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:38,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:39,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:40,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:41,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:43,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:43,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:45,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:45,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:47,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:49,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:49,825 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:47:49,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:49,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:47:51,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:53,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:55,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:47:55,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:47:55,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:57,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:47:59,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:00,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:00,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:01,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:03,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:04,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:48:05,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:06,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:07,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:09,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:10,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:11,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:11,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:13,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:15,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:15,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:17,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:17,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:19,826 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:48:19,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:19,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:48:21,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:22,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:23,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:25,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:25,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:27,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:28,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:29,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:30,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:31,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:33,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:33,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:34,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:48:35,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:37,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:39,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:39,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:40,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:41,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:43,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:44,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:45,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:45,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:47,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:49,829 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:48:49,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:49,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:48:50,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:51,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:53,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:55,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:48:55,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:56,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:48:57,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:48:59,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:00,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:01,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:01,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:03,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:04,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:49:05,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:07,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:07,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:09,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:10,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:11,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:12,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:13,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:15,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:15,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:17,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:18,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:19,832 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:49:19,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:19,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:49:21,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:23,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:23,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:25,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:25,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:27,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:29,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:29,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:30,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:31,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:33,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:34,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:34,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:49:35,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:37,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:39,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:40,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:40,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:41,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:43,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:45,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:45,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:45,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:47,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:49,838 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:49:49,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:49:49,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:51,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:51,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:53,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:55,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:49:56,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:49:57,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:49:58,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:00,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:00,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:02,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:02,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:04,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:04,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:50:06,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:08,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:08,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:10,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:10,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:12,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:13,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:14,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:15,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:16,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:18,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:19,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:19,837 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:50:19,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:50:20,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:22,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:24,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:24,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:25,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:26,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:28,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:30,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:30,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:30,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:32,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:34,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:34,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:50:35,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:36,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:38,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:40,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:40,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:41,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:42,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:44,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:45,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:46,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:46,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:48,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:49,839 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:50:49,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:50:50,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:52,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:52,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:54,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:55,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:50:56,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:50:57,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:50:58,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:00,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:00,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:02,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:03,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:04,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:04,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:51:06,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:08,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:09,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:10,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:10,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:12,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:14,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:14,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:15,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:16,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:18,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:19,842 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:51:19,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:19,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:51:20,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:22,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:24,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:25,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:25,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:26,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:28,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:30,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:30,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:31,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:32,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:34,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:34,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:51:36,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:36,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:38,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:40,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:40,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:42,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:42,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:44,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:45,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:46,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:47,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:48,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:49,847 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:51:49,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:51:50,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:52,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:53,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:51:54,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:55,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:51:56,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:58,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:51:58,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:00,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:00,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:02,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:04,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:04,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:04,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:52:06,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:08,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:09,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:10,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:10,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:12,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:14,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:15,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:15,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:16,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:18,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:19,848 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:52:19,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:52:20,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:21,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:22,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:24,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:25,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:26,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:26,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:28,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:30,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:30,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:32,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:32,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:34,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:34,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:52:36,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:37,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:38,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:40,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:40,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:42,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:43,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:44,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:45,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:46,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:48,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:48,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:49,851 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:52:49,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:52:50,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:52,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:54,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:54,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:52:55,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:52:56,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:58,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:52:59,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:00,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:00,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:02,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:04,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:04,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:53:05,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:06,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:08,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:10,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:10,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:10,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:12,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:14,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:15,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:16,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:16,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:18,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:19,853 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:53:19,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:53:20,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:21,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:22,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:24,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:25,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:26,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:27,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:28,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:30,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:30,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:32,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:33,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:34,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:34,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:53:36,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:38,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:38,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:40,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:40,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:42,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:44,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:44,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:45,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:46,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:48,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:49,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:49,856 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:53:49,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:53:50,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:52,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:54,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:55,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:53:55,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:53:56,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:53:58,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:00,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:00,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:00,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:02,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:04,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:04,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:54:06,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:06,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:08,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:10,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:10,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:11,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:12,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:14,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:15,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:16,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:17,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:18,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:19,859 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:54:19,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:54:20,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:22,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:22,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:24,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:25,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:26,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:28,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:28,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:30,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:30,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:32,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:33,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:34,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:34,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:54:36,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:38,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:39,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:40,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:40,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:42,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:44,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:45,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:45,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:46,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:48,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:49,862 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:54:49,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:54:50,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:50,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:52,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:54,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:55,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:54:56,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:54:56,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:54:58,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:00,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:00,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:01,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:02,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:04,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:04,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:55:06,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:07,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:08,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:10,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:10,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:12,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:12,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:14,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:15,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:16,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:18,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:18,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:19,865 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:55:19,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:55:20,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:22,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:23,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:24,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:25,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:26,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:28,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:29,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:30,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:30,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:32,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:34,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:34,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:34,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:55:36,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:38,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:40,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:40,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:40,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:42,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:44,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:45,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:45,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:46,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:48,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:49,867 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:55:49,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:55:50,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:51,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:52,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:54,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:55,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:55:56,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:55:57,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:55:58,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:00,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:00,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:02,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:02,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:04,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:04,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:56:06,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:08,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:08,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:10,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:10,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:12,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:13,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:14,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:15,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:16,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:18,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:19,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:19,870 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:56:19,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:56:20,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:22,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:24,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:24,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:25,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:26,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:28,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:30,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:30,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:30,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:32,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:34,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:34,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:56:35,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:36,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:38,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:40,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:40,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:41,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:42,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:44,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:45,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:46,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:46,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:48,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:49,873 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:56:49,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:56:50,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:52,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:52,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:54,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:55,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:56:56,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:56:58,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:56:58,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:00,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:00,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:02,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:03,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:04,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:04,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:57:06,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:08,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:09,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:10,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:10,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:12,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:14,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:14,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:15,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:16,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:18,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:19,875 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:57:19,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:19,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:57:20,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:22,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:24,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:25,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:25,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:26,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:28,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:30,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:30,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:31,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:32,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:34,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:34,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:57:36,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:36,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:38,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:40,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:40,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:42,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:42,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:44,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:45,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:46,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:47,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:48,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:49,878 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:57:49,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:57:50,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:52,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:53,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:57:54,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:55,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:57:56,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:58,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:57:58,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:00,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:00,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:02,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:04,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:04,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:04,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:58:06,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:08,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:10,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:10,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:10,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:12,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:14,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:15,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:15,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:16,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:18,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:19,881 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:58:19,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:58:20,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:21,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:22,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:24,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:25,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:26,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:26,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:28,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:30,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:30,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:32,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:32,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:34,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:34,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:58:36,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:37,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:38,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:40,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:40,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:42,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:43,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:44,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:45,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:46,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:48,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:48,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:49,884 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:58:49,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:58:50,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:52,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:54,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:58:54,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:55,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:58:56,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:58,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:58:59,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:00,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:00,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:02,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:04,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:04,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:59:05,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:06,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:08,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:10,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:10,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:11,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:12,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:14,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:15,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:16,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:16,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:18,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:19,888 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:59:19,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:59:20,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:22,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:22,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:24,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:25,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:26,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:27,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:28,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:30,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:30,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:32,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:33,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:34,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:34,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:59:36,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:38,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:38,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:40,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:40,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:42,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:44,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:44,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:45,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:46,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:48,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:49,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:49,889 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 00:59:49,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 00:59:50,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:52,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:54,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:55,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 00:59:55,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 00:59:56,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 00:59:58,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:00,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:00,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:00,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:02,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:04,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:04,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:00:06,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:06,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:08,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:10,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:10,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:11,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:12,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:14,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:15,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:16,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:17,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:18,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:19,892 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:00:19,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:00:20,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:22,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:23,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:24,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:25,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:26,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:28,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:28,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:30,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:30,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:32,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:34,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:34,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:34,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:00:36,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:38,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:39,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:40,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:40,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:42,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:44,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:45,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:45,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:46,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:48,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:49,895 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:00:49,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:00:50,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:50,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:52,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:54,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:55,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:00:56,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:00:56,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:00:58,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:00,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:00,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:01,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:02,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:04,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:04,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:01:06,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:07,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:08,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:10,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:10,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:12,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:12,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:14,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:15,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:16,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:18,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:18,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:19,901 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:01:19,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:01:20,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:22,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:24,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:24,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:25,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:26,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:28,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:29,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:30,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:30,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:32,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:34,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:34,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:01:35,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:36,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:38,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:40,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:40,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:40,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:42,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:44,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:45,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:46,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:46,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:48,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:49,901 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:01:49,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:01:50,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:51,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:52,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:54,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:55,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:01:56,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:01:57,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:01:58,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:00,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:00,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:02,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:02,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:04,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:04,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:02:06,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:08,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:08,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:10,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:10,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:12,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:13,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:14,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:15,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:16,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:18,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:19,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:19,904 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:02:19,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:02:20,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:22,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:24,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:24,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:25,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:26,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:28,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:30,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:30,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:30,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:32,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:34,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:34,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:02:36,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:36,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:38,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:40,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:40,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:41,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:42,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:44,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:45,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:46,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:47,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:49,906 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:02:49,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:02:52,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:52,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:54,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:55,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:02:56,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:02:57,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:02:58,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:00,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:00,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:02,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:02,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:04,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:04,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:03:06,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:08,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:08,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:10,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:10,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:12,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:14,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:14,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:15,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:16,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:18,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:19,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:19,914 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:03:19,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:03:20,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:22,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:24,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:25,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:25,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:26,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:28,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:30,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:30,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:34,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:34,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:03:35,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:36,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:38,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:40,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:40,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:40,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:42,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:44,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:45,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:46,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:46,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:48,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:49,912 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:03:49,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:03:50,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:51,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:52,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:54,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:55,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:03:56,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:03:57,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:03:58,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:00,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:00,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:02,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:02,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:04,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:04,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:04:06,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:08,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:08,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:10,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:10,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:12,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:13,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:14,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:15,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:16,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:18,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:19,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:19,915 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:04:19,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:04:20,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:22,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:24,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:24,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:25,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:26,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:28,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:30,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:30,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:30,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:32,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:34,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:34,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:04:35,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:36,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:38,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:40,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:40,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:41,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:42,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:44,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:45,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:46,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:47,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:48,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:49,917 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:04:49,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:04:50,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:52,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:52,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:54,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:55,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:04:56,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:04:58,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:04:58,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:00,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:00,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:02,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:03,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:04,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:04,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:05:06,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:08,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:09,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:10,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:10,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:12,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:14,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:14,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:15,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:16,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:18,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:19,920 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:05:19,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:19,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:05:20,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:22,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:24,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:25,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:25,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:26,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:28,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:30,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:30,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:31,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:32,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:34,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:34,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:05:36,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:36,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:38,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:40,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:40,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:42,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:42,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:44,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:45,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:46,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:47,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:48,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:49,922 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:05:49,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:05:50,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:52,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:53,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:54,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:55,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:05:56,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:05:58,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:05:58,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:00,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:00,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:02,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:04,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:04,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:04,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:06:06,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:08,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:10,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:10,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:10,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:12,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:14,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:15,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:15,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:17,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:19,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:19,925 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:06:19,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:06:21,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:21,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:23,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:25,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:25,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:26,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:27,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:29,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:30,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:31,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:32,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:33,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:34,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:06:35,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:37,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:37,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:39,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:40,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:41,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:43,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:43,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:45,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:45,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:47,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:48,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:49,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:49,928 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:06:49,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:06:51,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:53,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:54,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:06:55,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:55,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:06:57,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:59,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:06:59,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:00,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:01,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:03,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:04,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:07:05,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:05,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:07,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:09,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:10,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:10,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:11,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:13,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:15,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:15,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:16,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:17,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:19,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:19,930 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:07:19,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:07:21,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:22,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:23,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:25,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:25,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:27,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:27,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:29,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:30,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:31,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:33,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:33,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:34,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:07:35,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:37,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:38,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:39,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:40,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:41,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:43,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:44,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:45,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:45,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:47,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:49,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:49,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:49,933 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:07:49,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:07:51,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:53,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:55,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:55,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:07:55,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:07:57,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:07:59,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:00,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:00,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:01,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:03,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:04,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:08:05,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:06,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:07,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:09,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:10,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:11,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:11,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:13,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:15,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:15,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:17,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:17,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:19,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:19,937 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:08:19,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:08:21,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:23,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:23,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:25,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:25,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:27,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:28,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:29,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:30,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:31,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:33,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:34,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:34,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:08:35,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:37,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:39,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:39,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:40,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:41,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:43,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:45,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:45,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:45,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:47,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:49,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:49,947 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:08:49,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:08:50,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:51,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:53,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:55,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:55,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:08:56,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:08:57,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:08:59,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:00,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:00,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:01,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:09:01,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:03,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:04,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:09:05,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:06,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:07,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:09,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:10,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:11,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:11,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:13,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:15,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:15,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:17,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:17,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:19,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:19,942 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:09:19,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:09:21,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:22,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:23,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:25,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:25,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:27,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:28,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:29,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:30,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:31,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:33,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:33,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:34,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:09:35,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:37,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:39,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:39,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:40,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:41,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:43,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:44,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:45,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:45,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:47,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:49,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:49,945 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:09:49,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:49,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:09:51,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:53,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:55,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:09:55,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:55,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:09:57,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:09:59,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:00,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:00,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:01,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:03,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:04,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:10:05,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:05,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:07,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:09,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:10,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:11,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:11,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:13,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:15,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:15,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:16,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:17,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:19,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:19,948 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:10:19,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:10:21,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:22,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:23,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:25,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:25,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:27,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:27,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:29,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:30,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:31,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:33,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:33,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:34,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:10:35,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:37,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:39,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:39,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:40,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:42,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:44,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:44,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:45,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:46,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:48,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:49,959 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:10:49,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:49,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:10:50,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:52,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:54,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:55,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:10:55,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:10:56,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:10:58,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:00,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:00,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:01,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:02,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:04,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:04,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:11:06,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:06,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:08,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:10,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:10,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:12,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:12,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:14,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:15,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:16,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:17,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:18,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:19,954 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:11:19,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:11:20,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:22,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:23,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:24,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:25,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:26,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:28,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:28,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:30,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:30,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:32,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:34,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:34,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:34,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:11:36,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:38,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:39,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:40,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:40,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:42,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:44,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:45,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:45,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:46,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:48,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:49,956 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:11:49,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:11:50,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:51,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:52,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:54,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:55,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:11:56,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:11:56,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:11:58,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:00,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:00,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:02,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:02,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:04,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:04,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:12:06,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:07,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:08,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:10,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:10,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:12,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:13,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:14,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:15,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:16,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:18,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:18,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:19,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:19,959 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:12:19,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:12:22,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:24,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:24,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:25,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:26,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:28,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:29,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:30,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:30,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:32,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:34,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:34,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:12:35,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:36,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:38,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:40,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:40,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:40,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:42,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:44,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:45,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:46,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:46,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:48,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:49,962 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:12:49,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:12:50,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:52,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:52,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:54,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:55,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:12:56,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:12:57,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:12:58,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:00,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:00,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:02,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:03,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:04,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:04,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:13:06,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:08,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:08,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:10,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:10,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:12,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:14,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:14,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:15,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:16,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:18,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:19,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:19,964 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:13:19,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:13:20,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:22,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:24,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:25,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:25,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:26,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:28,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:30,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:30,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:30,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:32,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:34,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:34,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:13:36,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:36,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:38,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:40,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:40,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:41,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:42,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:44,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:45,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:46,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:47,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:48,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:49,967 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:13:49,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:13:50,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:52,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:52,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:13:54,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:55,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:13:56,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:58,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:13:58,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:00,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:00,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:02,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:04,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:04,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:04,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:14:06,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:08,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:09,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:10,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:10,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:12,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:14,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:15,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:15,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:16,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:18,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:19,970 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:14:19,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:14:20,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:20,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:22,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:24,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:25,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:26,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:26,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:28,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:30,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:30,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:31,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:32,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:34,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:34,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:14:36,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:37,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:38,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:40,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:40,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:42,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:42,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:44,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:45,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:46,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:48,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:48,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:49,973 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:14:49,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:14:50,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:52,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:53,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:14:54,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:55,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:14:56,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:58,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:14:59,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:00,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:00,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:02,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:04,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:04,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:15:05,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:06,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:08,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:10,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:10,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:10,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:12,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:14,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:15,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:16,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:16,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:18,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:19,975 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:15:19,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:15:20,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:21,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:22,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:24,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:25,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:26,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:27,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:28,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:30,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:30,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:32,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:32,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:34,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:34,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:15:36,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:38,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:38,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:40,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:40,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:42,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:43,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:44,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:45,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:46,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:48,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:49,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:49,978 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:15:49,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:15:50,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:52,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:54,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:54,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:15:55,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:15:56,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:15:58,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:00,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:00,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:00,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:02,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:04,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:04,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:16:05,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:06,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:08,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:10,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:10,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:11,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:12,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:14,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:15,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:16,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:17,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:18,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:19,984 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:16:19,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:16:20,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:22,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:22,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:24,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:25,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:26,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:28,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:28,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:30,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:30,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:32,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:33,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:34,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:34,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:16:36,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:38,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:39,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:40,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:40,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:42,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:44,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:44,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:45,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:46,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:48,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:49,984 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:16:49,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:49,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:16:50,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:52,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:54,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:55,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:16:55,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:16:56,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:16:58,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:00,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:00,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:01,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:02,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:04,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:04,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:17:06,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:06,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:08,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:10,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:10,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:12,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:12,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:14,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:15,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:16,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:18,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:18,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:19,987 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:17:19,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:17:20,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:22,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:23,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:24,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:25,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:26,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:28,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:29,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:30,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:30,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:32,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:34,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:34,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:34,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:17:36,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:38,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:40,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:40,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:40,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:42,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:44,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:45,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:45,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:46,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:48,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:49,990 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:17:49,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:17:50,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:51,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:52,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:54,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:55,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:17:56,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:17:56,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:17:58,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:00,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:00,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:02,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:02,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:04,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:04,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:18:06,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:07,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:08,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:10,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:10,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:12,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:13,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:14,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:15,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:16,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:18,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:18,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:19,994 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:18:19,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:18:20,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:22,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:24,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:24,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:25,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:26,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:28,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:29,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:30,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:30,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:32,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:34,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:34,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:18:35,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:36,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:38,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:40,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:40,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:41,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:42,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:44,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:45,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:46,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:46,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:48,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:49,995 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:18:49,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:18:50,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:52,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:52,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:54,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:55,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:18:56,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:18:57,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:18:58,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:00,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:00,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:02,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:03,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:04,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:04,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:19:06,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:08,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:08,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:10,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:10,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:12,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:13,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:14,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:15,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:16,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:18,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:19,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:19,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:19:19,998 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:19:20,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:22,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:24,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:24,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:25,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:26,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:28,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:29,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:30,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:30,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:32,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:34,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:34,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:34,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:19:36,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:38,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:40,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:40,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:40,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:42,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:44,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:45,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:46,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:46,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:19:49,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:19:50,001 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:19:52,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:19:55,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:19:57,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:00,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:02,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:02,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:20:02,067 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 01:20:02,074 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 01:20:02,083 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 01:20:02,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 01:20:04,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:05,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:20:07,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:10,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:12,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:14,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:16,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:17,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:20,003 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:20:20,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:20:23,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:24,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:20:24,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:20:25,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 01:20:25,085 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 01:20:25,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:20:25,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_5_1a5c30fa6dafb36e5691.table.json 2025-09-18 01:20:25,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 01:20:26,033 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmp1sn08_jd 2025-09-18 01:20:26,238 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/y4743kf0-media/table/rollouts_5_1a5c30fa6dafb36e5691.table.json 2025-09-18 01:20:26,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:26,856 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTcxODc0MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTYwMDA2MQ==', 'versionIndex': 3}}} 2025-09-18 01:20:26,856 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 01:20:26,856 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_5_1a5c30fa6dafb36e5691.table.json with policy now 2025-09-18 01:20:26,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:27,165 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/dpiprric-media/table/rollouts_5_1a5c30fa6dafb36e5691.table.json 2025-09-18 01:20:28,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:28,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:30,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:31,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:32,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:33,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:34,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:35,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:20:36,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:38,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:39,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:40,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:40,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:42,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:44,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:45,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:45,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:46,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:48,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:50,007 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:20:50,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:20:50,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:50,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:52,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:54,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:55,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:20:56,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:20:56,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:20:58,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:00,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:00,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:01,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:02,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:04,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:05,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:21:06,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:07,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:08,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:10,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:10,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:12,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:12,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:14,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:15,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:16,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:18,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:18,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:20,008 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:21:20,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:21:20,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:22,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:23,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:24,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:25,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:26,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:28,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:29,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:30,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:30,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:32,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:34,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:34,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:35,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:21:36,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:38,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:40,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:40,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:40,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:42,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:44,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:45,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:45,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:46,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:48,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:50,011 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:21:50,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:21:50,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:51,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:52,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:54,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:55,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:21:56,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:21:57,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:21:58,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:00,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:00,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:02,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:02,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:04,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:05,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:22:06,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:08,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:08,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:10,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:10,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:12,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:13,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:14,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:15,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:16,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:18,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:19,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:20,014 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:22:20,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:22:20,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:23,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:24,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:25,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:25,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:27,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:29,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:30,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:30,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:31,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:33,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:35,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:35,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:22:35,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:37,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:39,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:40,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:41,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:41,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:43,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:45,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:45,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:46,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:47,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:49,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:50,021 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:22:50,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:22:51,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:52,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:53,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:55,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:55,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:22:57,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:22:58,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:22:59,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:00,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:01,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:03,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:03,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:05,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:05,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:23:07,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:09,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:09,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:10,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:11,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:13,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:14,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:15,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:15,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:17,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:19,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:20,020 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:23:20,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:20,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:23:21,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:23,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:25,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:25,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:25,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:27,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:29,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:30,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:31,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:31,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:33,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:35,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:35,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:23:36,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:37,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:39,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:40,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:41,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:42,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:43,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:45,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:45,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:47,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:47,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:49,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:50,024 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:23:50,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:23:51,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:53,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:53,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:55,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:55,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:23:57,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:23:59,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:23:59,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:00,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:01,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:03,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:04,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:05,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:05,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:24:07,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:09,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:10,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:10,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:11,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:13,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:15,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:15,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:15,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:17,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:19,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:20,026 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:24:20,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:24:21,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:21,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:23,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:25,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:25,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:26,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:27,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:29,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:30,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:31,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:32,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:33,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:35,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:35,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:24:37,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:37,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:39,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:40,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:41,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:43,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:43,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:45,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:45,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:47,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:48,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:49,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:50,029 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:24:50,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:24:51,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:53,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:54,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:24:55,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:55,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:24:57,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:59,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:24:59,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:00,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:01,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:03,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:05,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:05,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:25:05,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:07,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:09,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:10,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:10,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:11,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:13,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:15,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:15,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:16,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:17,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:19,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:20,032 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:25:20,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:25:21,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:22,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:23,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:25,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:25,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:27,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:27,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:29,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:30,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:31,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:33,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:33,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:35,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:35,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:25:37,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:38,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:39,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:40,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:41,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:43,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:44,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:45,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:45,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:47,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:49,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:49,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:50,035 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:25:50,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:25:51,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:53,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:55,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:55,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:25:55,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:25:57,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:25:59,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:00,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:00,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:01,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:03,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:05,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:05,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:26:06,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:07,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:09,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:10,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:11,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:11,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:13,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:15,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:15,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:17,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:17,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:19,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:20,038 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:26:20,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:26:21,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:23,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:23,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:25,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:25,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:27,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:28,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:29,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:30,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:31,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:33,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:34,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:35,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:35,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:26:37,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:39,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:39,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:40,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:41,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:43,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:45,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:45,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:45,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:47,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:49,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:50,040 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:26:50,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:26:51,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:51,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:53,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:55,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:55,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:26:56,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:26:57,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:26:59,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:00,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:01,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:01,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:03,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:05,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:05,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:27:07,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:07,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:09,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:10,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:11,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:12,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:13,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:15,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:15,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:17,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:18,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:19,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:20,043 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:27:20,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:27:21,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:23,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:24,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:25,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:25,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:27,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:29,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:29,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:30,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:31,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:33,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:35,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:35,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:27:35,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:37,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:39,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:40,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:40,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:41,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:43,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:45,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:45,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:46,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:47,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:49,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:50,046 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:27:50,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:27:51,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:51,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:53,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:55,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:55,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:27:57,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:27:57,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:27:59,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:00,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:01,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:02,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:03,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:05,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:28:05,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:07,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:08,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:09,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:10,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:11,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:13,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:13,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:15,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:15,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:17,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:19,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:19,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:20,049 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:28:20,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:28:21,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:23,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:24,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:25,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:25,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:27,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:29,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:30,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:30,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:31,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:33,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:35,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:28:35,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:36,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:37,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:39,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:40,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:41,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:41,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:43,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:45,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:45,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:47,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:47,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:49,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:50,052 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:28:50,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:28:51,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:52,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:53,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:55,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:55,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:28:57,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:28:58,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:28:59,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:00,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:01,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:03,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:03,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:05,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:29:05,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:07,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:09,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:09,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:10,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:11,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:13,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:14,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:15,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:15,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:17,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:19,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:20,055 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:29:20,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:20,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:29:21,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:23,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:25,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:25,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:25,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:27,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:29,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:30,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:31,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:31,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:33,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:35,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:29:35,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:36,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:37,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:39,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:40,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:41,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:42,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:43,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:45,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:45,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:47,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:48,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:49,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:50,057 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:29:50,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:29:51,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:53,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:53,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:55,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:55,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:29:57,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:29:59,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:29:59,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:00,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:01,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:03,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:04,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:05,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:30:05,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:07,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:09,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:10,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:10,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:11,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:13,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:15,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:15,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:15,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:17,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:19,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:20,063 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:30:20,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:30:21,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:21,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:23,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:25,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:25,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:26,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:27,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:29,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:30,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:31,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:32,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:33,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:35,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:30:35,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:37,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:37,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:39,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:40,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:41,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:43,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:43,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:45,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:45,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:47,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:48,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:49,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:50,064 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:30:50,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:30:51,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:53,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:54,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:30:55,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:55,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:30:57,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:30:59,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:00,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:00,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:01,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:03,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:05,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:31:05,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:05,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:07,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:09,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:10,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:11,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:11,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:13,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:15,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:15,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:16,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:17,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:19,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:20,066 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:31:20,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:31:21,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:22,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:23,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:25,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:25,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:27,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:27,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:29,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:30,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:31,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:33,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:33,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:35,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:31:35,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:37,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:38,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:39,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:40,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:41,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:43,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:44,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:45,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:45,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:47,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:49,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:49,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:50,069 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:31:50,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:31:51,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:53,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:55,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:31:55,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:55,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:31:57,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:31:59,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:00,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:00,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:01,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:03,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:05,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:32:05,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:06,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:07,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:09,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:10,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:11,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:12,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:13,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:15,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:15,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:17,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:17,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:19,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:20,074 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:32:20,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:32:21,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:23,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:23,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:25,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:25,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:27,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:28,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:29,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:30,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:31,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:33,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:34,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:35,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:32:35,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:37,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:39,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:39,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:40,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:41,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:43,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:45,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:45,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:45,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:47,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:49,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:50,075 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:32:50,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:32:51,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:51,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:53,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:55,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:55,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:32:56,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:32:57,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:32:59,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:00,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:01,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:01,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:03,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:05,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:33:05,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:07,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:07,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:09,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:10,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:11,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:13,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:13,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:15,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:15,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:17,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:18,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:19,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:20,077 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:33:20,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:33:21,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:23,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:24,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:25,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:25,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:27,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:28,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:28,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:29,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:29,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:29,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:33:29,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:29,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:30,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:31,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:33,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:34,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:35,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:33:35,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:37,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:39,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:39,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:40,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:41,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:43,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:44,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:45,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:45,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:47,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:49,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:50,080 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:33:50,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:50,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:33:51,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:53,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:55,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:33:55,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:55,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:33:57,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:33:59,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:00,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:00,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:01,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:03,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:05,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:34:05,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:05,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:07,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:09,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:10,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:11,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:11,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:13,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:15,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:15,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:16,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:17,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:19,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:20,086 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:34:20,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:34:21,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:22,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:23,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:25,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:25,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:27,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:27,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:29,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:30,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:31,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:33,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:33,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:35,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:34:35,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:37,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:39,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:39,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:40,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:41,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:43,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:44,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:45,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:45,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:47,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:49,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:50,087 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:34:50,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:50,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:34:51,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:53,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:55,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:34:55,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:55,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:34:57,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:34:59,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:00,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:01,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:01,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:03,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:05,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:35:05,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:06,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:07,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:09,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:10,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:11,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:12,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:13,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:15,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:15,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:17,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:17,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:19,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:20,090 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:35:20,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:35:21,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:23,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:23,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:25,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:25,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:27,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:28,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:29,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:30,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:31,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:33,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:34,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:35,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:35:35,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:37,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:39,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:39,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:40,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:41,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:43,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:45,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:45,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:45,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:47,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:49,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:50,093 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:35:50,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:35:51,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:51,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:53,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:55,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:55,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:35:56,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:35:57,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:35:59,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:00,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:01,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:02,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:03,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:05,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:36:05,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:07,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:07,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:09,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:10,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:11,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:13,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:13,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:15,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:15,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:17,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:18,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:19,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:20,098 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:36:20,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:36:21,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:23,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:24,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:25,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:25,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:27,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:29,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:29,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:30,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:31,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:33,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:35,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:36:35,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:35,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:37,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:39,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:40,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:40,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:41,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:43,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:45,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:45,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:46,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:47,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:49,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:50,098 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:36:50,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:36:51,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:51,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:53,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:55,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:55,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:36:57,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:36:57,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:36:59,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:00,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:01,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:03,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:03,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:05,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:37:05,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:07,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:08,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:09,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:10,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:11,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:13,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:14,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:15,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:15,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:17,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:19,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:19,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:20,101 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:37:20,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:37:21,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:23,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:25,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:25,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:25,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:27,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:29,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:30,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:30,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:31,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:33,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:35,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:37:35,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:36,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:37,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:39,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:40,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:41,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:41,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:43,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:45,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:45,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:47,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:47,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:49,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:50,105 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:37:50,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:37:51,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:52,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:53,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:55,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:37:55,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:57,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:37:58,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:37:59,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:00,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:01,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:03,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:04,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:05,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:38:05,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:07,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:09,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:09,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:10,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:11,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:13,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:15,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:15,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:15,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:17,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:19,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:20,108 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:38:20,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:38:21,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:21,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:23,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:25,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:25,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:26,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:27,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:29,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:30,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:31,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:32,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:34,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:35,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:38:36,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:37,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:38,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:40,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:40,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:42,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:42,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:44,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:45,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:46,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:48,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:48,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:50,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:50,111 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:38:50,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:38:52,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:53,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:38:54,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:55,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:38:56,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:58,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:38:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:00,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:00,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:02,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:04,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:05,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:05,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:39:06,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:08,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:10,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:10,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:10,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:12,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:14,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:15,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:16,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:16,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:18,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:20,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:20,113 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:39:20,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:39:21,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:22,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:24,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:25,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:26,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:27,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:28,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:30,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:30,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:32,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:32,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:34,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:35,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:39:36,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:38,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:38,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:40,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:40,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:42,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:43,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:44,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:45,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:46,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:48,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:49,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:50,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:50,116 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:39:50,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:39:52,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:54,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:54,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:39:55,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:39:56,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:39:58,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:00,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:00,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:00,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:02,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:04,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:05,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:40:05,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:06,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:08,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:10,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:10,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:11,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:12,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:14,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:15,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:16,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:17,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:18,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:20,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:20,119 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:40:20,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:40:22,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:22,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:24,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:25,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:26,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:28,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:28,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:30,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:30,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:32,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:33,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:34,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:35,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:40:36,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:38,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:39,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:40,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:40,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:42,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:44,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:44,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:45,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:46,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:48,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:50,122 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:40:50,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:50,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:50,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:40:52,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:54,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:55,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:40:55,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:40:56,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:40:58,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:00,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:00,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:01,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:02,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:04,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:05,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:41:06,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:06,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:08,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:10,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:10,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:12,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:12,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:14,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:15,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:16,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:17,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:18,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:20,125 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:41:20,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:20,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:41:22,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:23,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:24,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:25,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:26,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:28,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:29,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:30,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:30,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:32,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:34,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:34,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:35,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:41:36,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:38,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:40,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:40,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:40,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:42,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:44,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:45,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:45,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:46,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:48,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:50,128 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:41:50,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:50,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:41:51,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:52,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:54,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:55,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:41:56,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:41:56,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:41:58,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:00,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:00,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:02,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:02,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:04,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:05,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:42:06,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:07,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:08,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:10,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:10,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:12,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:13,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:14,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:15,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:16,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:18,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:18,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:20,129 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:42:20,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:20,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:42:22,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:24,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:24,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:25,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:26,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:28,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:30,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:30,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:30,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:32,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:34,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:35,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:42:35,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:36,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:38,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:40,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:40,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:41,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:42,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:44,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:45,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:46,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:46,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:48,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:50,133 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:42:50,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:50,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:42:52,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:52,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:54,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:55,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:42:56,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:42:57,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:42:58,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:00,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:00,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:02,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:03,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:04,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:05,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:43:06,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:08,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:08,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:10,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:10,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:12,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:14,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:14,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:15,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:16,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:18,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:19,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:20,135 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:43:20,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:20,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:43:22,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:24,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:25,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:25,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:26,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:28,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:30,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:30,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:30,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:32,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:34,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:35,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:43:36,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:36,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:38,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:40,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:40,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:41,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:42,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:44,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:45,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:46,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:47,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:48,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:50,140 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:43:50,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:50,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:43:52,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:53,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:43:54,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:56,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:43:56,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:58,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:43:58,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:00,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:01,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:02,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:04,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:04,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:05,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:44:06,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:08,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:09,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:10,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:11,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:12,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:14,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:15,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:16,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:16,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:18,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:20,140 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:44:20,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:44:20,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:21,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:22,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:24,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:26,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:26,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:26,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:28,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:30,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:31,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:31,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:32,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:34,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:35,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:44:36,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:37,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:38,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:40,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:41,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:42,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:42,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:44,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:46,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:46,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:48,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:48,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:50,143 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:44:50,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:44:50,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:52,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:54,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:44:54,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:56,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:44:56,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:58,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:44:59,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:00,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:01,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:02,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:04,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:05,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:05,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:45:06,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:08,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:10,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:10,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:11,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:12,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:14,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:16,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:16,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:16,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:18,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:20,147 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:45:20,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:45:20,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:21,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:22,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:24,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:26,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:26,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:27,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:28,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:30,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:31,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:32,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:32,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:34,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:35,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:45:36,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:38,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:38,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:40,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:41,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:42,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:43,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:44,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:46,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:46,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:48,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:49,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:50,152 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:45:50,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:45:50,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:52,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:54,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:55,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:45:56,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:45:56,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:45:58,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:00,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:00,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:01,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:02,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:04,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:05,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:46:06,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:06,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:08,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:10,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:11,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:11,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:12,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:14,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:16,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:16,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:17,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:18,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:20,153 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:46:20,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:46:20,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:22,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:22,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:24,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:26,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:26,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:28,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:28,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:30,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:31,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:32,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:33,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:34,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:35,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:46:36,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:38,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:39,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:40,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:41,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:42,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:44,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:44,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:46,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:46,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:48,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:50,155 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:46:50,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:50,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:46:50,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:52,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:54,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:55,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:46:56,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:46:56,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:46:58,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:00,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:01,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:01,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:02,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:04,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:05,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:47:06,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:07,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:08,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:10,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:11,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:12,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:12,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:14,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:16,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:16,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:18,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:18,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:20,158 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:47:20,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:47:20,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:22,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:23,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:24,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:26,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:26,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:28,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:29,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:30,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:31,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:32,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:34,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:34,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:35,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:47:36,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:38,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:40,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:40,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:41,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:42,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:44,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:45,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:46,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:46,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:48,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:50,161 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:47:50,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:47:50,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:51,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:52,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:54,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:56,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:47:56,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:47:56,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:47:58,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:00,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:01,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:02,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:02,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:04,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:05,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:48:06,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:07,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:08,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:10,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:11,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:12,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:13,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:14,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:16,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:16,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:18,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:19,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:20,164 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:48:20,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:48:20,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:22,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:24,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:24,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:26,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:26,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:28,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:30,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:30,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:31,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:32,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:34,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:35,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:48:35,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:36,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:38,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:40,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:41,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:41,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:42,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:44,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:46,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:46,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:46,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:48,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:50,168 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:48:50,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:48:50,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:52,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:52,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:54,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:56,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:48:56,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:48:57,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:48:58,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:00,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:01,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:02,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:03,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:04,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:05,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:49:06,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:08,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:08,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:10,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:11,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:12,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:14,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:14,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:16,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:16,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:18,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:19,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:20,172 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:49:20,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:49:20,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:22,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:24,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:25,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:26,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:26,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:28,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:30,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:31,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:31,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:32,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:34,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:35,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:49:36,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:36,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:38,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:40,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:41,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:42,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:42,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:44,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:46,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:46,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:47,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:48,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:50,172 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:49:50,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:49:50,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:52,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:53,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:49:54,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:56,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:49:56,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:58,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:49:58,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:00,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:01,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:02,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:04,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:04,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:05,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:50:06,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:08,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:09,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:10,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:11,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:12,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:14,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:15,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:16,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:16,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:18,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:20,176 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:50:20,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:50:20,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:21,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:22,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:24,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:26,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:26,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:26,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:28,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:30,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:31,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:32,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:32,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:34,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:35,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:50:36,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:37,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:38,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:40,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:41,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:42,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:43,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:44,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:46,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:46,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:48,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:48,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:50,178 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:50:50,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:50:50,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:52,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:54,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:50:54,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:56,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:50:56,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:58,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:50:59,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:00,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:01,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:02,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:04,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:05,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:05,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:51:06,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:08,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:10,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:10,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:11,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:12,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:14,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:16,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:16,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:16,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:18,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:20,183 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:51:20,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:51:20,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:21,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:22,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:24,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:26,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:26,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:27,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:28,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:30,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:31,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:32,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:32,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:34,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:35,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:51:36,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:38,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:38,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:40,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:41,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:42,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:44,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:44,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:46,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:46,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:48,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:49,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:50,184 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:51:50,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:51:50,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:52,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:54,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:55,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:51:56,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:51:56,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:51:58,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:00,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:00,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:01,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:02,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:04,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:05,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:52:06,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:06,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:08,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:10,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:11,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:11,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:12,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:14,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:16,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:16,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:17,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:18,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:20,186 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:52:20,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:52:20,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:22,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:22,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:24,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:26,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:26,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:28,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:28,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:30,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:31,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:32,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:33,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:34,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:35,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:52:36,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:38,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:39,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:40,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:41,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:42,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:44,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:44,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:46,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:46,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:48,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:50,189 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:52:50,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:50,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:52:50,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:52,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:54,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:56,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:52:56,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:52:56,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:52:58,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:00,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:01,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:01,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:02,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:04,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:05,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:53:06,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:07,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:08,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:10,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:11,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:12,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:12,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:14,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:16,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:16,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:18,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:18,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:20,194 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:53:20,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:53:20,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:22,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:23,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:24,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:26,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:26,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:28,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:29,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:30,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:31,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:32,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:34,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:34,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:35,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:53:36,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:38,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:40,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:40,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:41,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:42,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:44,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:45,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:46,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:46,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:48,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:50,194 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:53:50,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:53:50,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:51,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:52,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:54,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:56,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:53:56,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:53:57,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:53:58,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:00,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:01,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:02,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:02,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:04,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:05,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:54:06,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:08,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:08,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:10,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:11,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:12,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:13,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:14,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:16,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:16,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:18,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:19,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:20,198 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:54:20,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:54:20,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:22,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:24,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:24,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:26,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:26,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:28,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:30,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:30,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:31,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:32,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:34,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:35,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:54:35,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:36,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:38,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:40,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:41,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:41,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:42,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:44,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:46,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:46,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:46,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:48,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:50,201 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:54:50,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:54:51,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:52,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:53,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:55,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:56,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:54:57,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:54:57,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:54:59,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:01,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:01,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:03,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:03,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:05,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:05,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:55:07,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:08,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:09,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:11,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:11,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:13,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:14,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:15,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:16,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:17,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:19,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:20,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:20,204 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:55:20,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:55:21,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:23,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:25,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:25,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:26,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:27,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:29,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:31,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:31,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:31,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:33,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:35,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:35,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:55:36,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:37,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:39,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:41,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:41,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:42,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:43,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:45,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:46,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:47,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:47,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:49,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:50,206 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:55:50,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:55:51,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:53,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:53,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:55,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:56,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:55:57,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:55:58,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:55:59,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:01,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:01,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:03,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:04,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:05,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:05,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:56:07,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:09,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:09,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:11,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:11,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:13,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:15,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:15,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:16,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:17,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:19,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:20,210 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:56:20,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:56:21,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:21,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:23,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:25,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:26,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:26,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:27,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:29,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:31,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:31,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:32,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:33,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:33,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:33,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,054 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:34,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 01:56:35,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:35,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:56:37,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:37,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:39,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:41,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:41,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:43,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:43,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:45,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:46,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:47,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:48,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:49,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:50,213 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:56:50,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:56:51,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:53,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:54,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:56:55,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:56,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:56:57,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:59,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:56:59,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:01,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:01,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:03,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:05,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:05,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:57:05,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:07,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:09,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:11,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:11,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:11,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:13,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:15,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:16,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:16,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:17,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:19,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:20,214 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:57:20,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:57:21,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:22,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:23,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:25,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:26,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:27,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:27,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:29,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:31,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:31,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:33,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:33,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:35,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:35,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:57:37,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:38,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:39,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:41,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:41,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:43,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:44,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:45,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:46,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:47,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:49,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:49,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:50,217 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:57:50,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:57:51,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:53,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:55,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:55,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:57:56,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:57:57,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:57:59,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:00,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:01,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:01,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:03,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:05,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:05,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:58:06,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:07,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:09,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:11,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:11,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:12,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:13,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:15,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:16,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:17,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:17,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:19,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:20,219 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:58:20,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:58:21,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:23,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:23,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:25,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:26,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:27,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:28,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:29,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:31,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:31,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:33,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:34,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:35,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:35,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:58:37,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:39,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:39,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:41,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:41,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:43,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:45,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:45,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:46,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:47,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:49,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:50,251 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:58:50,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:58:51,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:51,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:53,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:55,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:56,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:58:56,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:58:57,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:58:59,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:01,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:01,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:01,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:03,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:05,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:05,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:59:07,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:07,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:09,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:11,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:11,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:13,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:13,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:15,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:16,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:17,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:18,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:19,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:20,250 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:59:20,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:59:21,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:23,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:23,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:25,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:26,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:27,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:29,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:29,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:31,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:31,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:33,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:34,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:35,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:35,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:59:37,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:39,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:39,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:41,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:41,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:43,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:45,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:45,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:46,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:47,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:49,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:50,253 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 01:59:50,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:50,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 01:59:51,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:53,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:55,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:55,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 01:59:56,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 01:59:57,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 01:59:59,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:00,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:01,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:01,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:03,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:05,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:05,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:00:06,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:07,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:00:07,874 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 02:00:07,882 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 02:00:07,892 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 02:00:08,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 02:00:09,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:10,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:11,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:11,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:16,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:16,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:20,255 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:00:21,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:21,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:21,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:00:22,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:27,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:27,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:31,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:00:31,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:00:31,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 02:00:31,585 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 02:00:31,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:00:32,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_6_6e93dc5c1a3f79a3476b.table.json 2025-09-18 02:00:32,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 02:00:32,508 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpgn8t57sb 2025-09-18 02:00:32,720 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/2o86a2cb-media/table/rollouts_6_6e93dc5c1a3f79a3476b.table.json 2025-09-18 02:00:33,271 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTg1MDQ3OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTcxODc0MQ==', 'versionIndex': 4}}} 2025-09-18 02:00:33,272 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 02:00:33,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:33,272 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_6_6e93dc5c1a3f79a3476b.table.json with policy now 2025-09-18 02:00:33,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:33,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:33,665 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/t7xc7ftg-media/table/rollouts_6_6e93dc5c1a3f79a3476b.table.json 2025-09-18 02:00:35,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:36,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:00:37,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:39,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:39,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:41,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:41,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:43,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:44,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:45,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:46,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:47,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:49,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:50,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:50,258 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:00:51,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:00:51,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:53,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:55,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:55,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:00:56,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:00:57,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:00:59,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:01,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:01,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:01,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:03,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:05,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:06,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:01:06,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:07,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:09,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:11,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:11,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:12,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:13,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:15,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:16,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:17,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:17,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:19,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:20,264 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:01:21,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:01:21,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:23,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:23,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:25,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:26,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:27,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:28,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:29,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:31,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:31,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:33,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:34,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:35,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:36,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:01:37,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:39,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:39,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:41,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:41,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:43,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:45,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:45,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:46,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:47,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:49,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:50,265 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:01:51,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:01:51,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:51,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:53,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:55,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:56,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:01:56,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:01:57,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:01:59,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:01,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:01,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:02,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:03,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:05,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:06,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:02:07,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:07,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:09,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:11,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:11,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:13,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:13,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:15,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:16,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:17,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:18,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:19,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:20,267 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:02:21,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:02:21,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:23,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:24,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:25,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:26,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:27,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:29,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:29,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:31,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:31,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:33,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:35,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:35,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:36,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:02:37,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:39,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:40,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:41,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:41,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:43,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:45,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:46,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:46,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:47,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:49,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:50,270 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:02:51,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:02:51,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:53,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:55,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:56,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:02:56,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:02:57,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:02:59,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:01,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:01,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:02,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:03,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:05,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:06,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:03:07,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:07,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:09,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:11,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:11,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:13,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:13,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:15,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:16,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:17,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:18,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:19,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:20,276 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:03:21,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:03:21,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:23,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:24,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:25,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:26,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:27,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:29,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:29,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:31,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:31,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:33,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:35,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:35,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:36,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:03:37,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:39,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:40,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:41,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:41,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:43,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:45,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:46,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:46,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:47,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:49,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:50,276 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:03:51,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:03:51,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:52,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:53,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:55,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:56,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:03:57,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:03:57,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:03:59,338 WARNING FileStreamThread:1208566 [file_stream.py:request_with_retry():668] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/weatherbenching/ERA5_sfno/jdwsvk36/file_stream. func: functools.partial(>, timeout=180), args: ('https://api.wandb.ai/files/weatherbenching/ERA5_sfno/jdwsvk36/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 489, 'content': ['{"system.proc.memory.availableMB": 153144.39, "system.memory": 40.47, "system.proc.memory.rssMB": 7138.63, "system.proc.memory.percent": 2.78, "system.network.sent": 1697864172.73, "system.network.recv": 2413648684.33, "system.cpu": 1.2, "system.cpu.0.cpu_percent": 6.83, "system.cpu.1.cpu_percent": 8.83, "system.cpu.2.cpu_percent": 18.11, "system.cpu.3.cpu_percent": 4.79, "system.cpu.4.cpu_percent": 6.43, "system.cpu.5.cpu_percent": 34.95, "system.cpu.6.cpu_percent": 3.51, "system.cpu.7.cpu_percent": 2.11, "system.cpu.8.cpu_percent": 25.03, "system.cpu.9.cpu_percent": 5.4, "system.cpu.10.cpu_percent": 39.63, "system.cpu.11.cpu_percent": 1.94, "system.cpu.12.cpu_percent": 1.77, "system.cpu.13.cpu_percent": 26.01, "system.cpu.14.cpu_percent": 2.04, "system.cpu.15.cpu_percent": 0.83, "system.cpu.16.cpu_percent": 3.36, "system.cpu.17.cpu_percent": 4.95, "system.cpu.18.cpu_percent": 14.19, "system.cpu.19.cpu_percent": 7.46, "system.cpu.20.cpu_percent": 7.51, "system.cpu.21.cpu_percent": 2.04, "system.cpu.22.cpu_percent": 1.47, "system.cpu.23.cpu_percent": 25.51, "system.cpu.24.cpu_percent": 29.08, "system.cpu.25.cpu_percent": 10.23, "system.cpu.26.cpu_percent": 4.63, "system.cpu.27.cpu_percent": 8.9, "system.cpu.28.cpu_percent": 6.56, "system.cpu.29.cpu_percent": 15.86, "system.cpu.30.cpu_percent": 4.99, "system.cpu.31.cpu_percent": 0.67, "system.cpu.32.cpu_percent": 6.05, "system.cpu.33.cpu_percent": 6.6, "system.cpu.34.cpu_percent": 4.87, "system.cpu.35.cpu_percent": 32.33, "system.cpu.36.cpu_percent": 2.54, "system.cpu.37.cpu_percent": 6.51, "system.cpu.38.cpu_percent": 1.27, "system.cpu.39.cpu_percent": 8.83, "system.cpu.40.cpu_percent": 21.87, "system.cpu.41.cpu_percent": 27.23, "system.cpu.42.cpu_percent": 11.29, "system.cpu.43.cpu_percent": 2.14, "system.cpu.44.cpu_percent": 20.41, "system.cpu.45.cpu_percent": 0.81, "system.cpu.46.cpu_percent": 0.77, "system.cpu.47.cpu_percent": 1.61, "system.cpu.48.cpu_percent": 2.74, "system.cpu.49.cpu_percent": 8.18, "system.cpu.50.cpu_percent": 9.17, "system.cpu.51.cpu_percent": 31.07, "system.cpu.52.cpu_percent": 2.07, "system.cpu.53.cpu_percent": 3.5, "system.cpu.54.cpu_percent": 3.31, "system.cpu.55.cpu_percent": 11.95, "system.cpu.56.cpu_percent": 39.54, "system.cpu.57.cpu_percent": 41.23, "system.cpu.58.cpu_percent": 12.1, "system.cpu.59.cpu_percent": 1.67, "system.cpu.60.cpu_percent": 5.09, "system.cpu.61.cpu_percent": 1.0, "system.cpu.62.cpu_percent": 1.13, "system.cpu.63.cpu_percent": 0.53, "system.cpu.64.cpu_percent": 3.4, "system.cpu.65.cpu_percent": 5.81, "system.cpu.66.cpu_percent": 9.41, "system.cpu.67.cpu_percent": 8.81, "system.cpu.68.cpu_percent": 3.53, "system.cpu.69.cpu_percent": 4.47, "system.cpu.70.cpu_percent": 2.01, "system.cpu.71.cpu_percent": 4.33, "system.cpu.72.cpu_percent": 15.91, "system.cpu.73.cpu_percent": 6.57, "system.cpu.74.cpu_percent": 11.33, "system.cpu.75.cpu_percent": 4.67, "system.cpu.76.cpu_percent": 3.01, "system.cpu.77.cpu_percent": 5.71, "system.cpu.78.cpu_percent": 3.71, "system.cpu.79.cpu_percent": 1.91, "system.cpu.80.cpu_percent": 20.46, "system.cpu.81.cpu_percent": 1.84, "system.cpu.82.cpu_percent": 2.57, "system.cpu.83.cpu_percent": 0.7, "system.cpu.84.cpu_percent": 2.17, "system.cpu.85.cpu_percent": 4.54, "system.cpu.86.cpu_percent": 0.97, "system.cpu.87.cpu_percent": 9.49, "system.cpu.88.cpu_percent": 16.61, "system.cpu.89.cpu_percent": 6.48, "system.cpu.90.cpu_percent": 5.05, "system.cpu.91.cpu_percent": 5.78, "system.cpu.92.cpu_percent": 1.65, "system.cpu.93.cpu_percent": 8.59, "system.cpu.94.cpu_percent": 3.17, "system.cpu.95.cpu_percent": 1.5, "system.cpu.96.cpu_percent": 11.53, "system.cpu.97.cpu_percent": 4.57, "system.cpu.98.cpu_percent": 3.75, "system.cpu.99.cpu_percent": 16.06, "system.cpu.100.cpu_percent": 2.63, "system.cpu.101.cpu_percent": 6.75, "system.cpu.102.cpu_percent": 1.91, "system.cpu.103.cpu_percent": 4.97, "system.cpu.104.cpu_percent": 13.41, "system.cpu.105.cpu_percent": 9.43, "system.cpu.106.cpu_percent": 3.19, "system.cpu.107.cpu_percent": 1.83, "system.cpu.108.cpu_percent": 4.79, "system.cpu.109.cpu_percent": 1.74, "system.cpu.110.cpu_percent": 1.57, "system.cpu.111.cpu_percent": 1.03, "system.cpu.112.cpu_percent": 5.34, "system.cpu.113.cpu_percent": 5.38, "system.cpu.114.cpu_percent": 7.1, "system.cpu.115.cpu_percent": 8.77, "system.cpu.116.cpu_percent": 2.71, "system.cpu.117.cpu_percent": 5.32, "system.cpu.118.cpu_percent": 7.59, "system.cpu.119.cpu_percent": 4.31, "system.cpu.120.cpu_percent": 5.83, "system.cpu.121.cpu_percent": 9.61, "system.cpu.122.cpu_percent": 1.17, "system.cpu.123.cpu_percent": 1.27, "system.cpu.124.cpu_percent": 2.34, "system.cpu.125.cpu_percent": 0.93, "system.cpu.126.cpu_percent": 1.4, "system.cpu.127.cpu_percent": 0.6, "system.proc.cpu.threads": 101, "system.disk.\\\\.usagePercent": 0.5, "system.disk.\\\\.usageGB": 0.58, "system.disk.in": 322.25, "system.disk.out": 0.03, "system.gpu.0.memoryAllocated": 92.82, "system.gpu.process.0.memoryAllocated": 92.82, "system.gpu.1.memoryAllocated": 92.02, "system.gpu.2.memoryAllocated": 92.05, "system.gpu.3.memoryAllocated": 93.42, "system.gpu.0.memoryAllocatedBytes": 79732408320.0, "system.gpu.process.0.memoryAllocatedBytes": 79732408320.0, "system.gpu.1.memoryAllocatedBytes": 79046639616.0, "system.gpu.2.memoryAllocatedBytes": 79071805440.0, "system.gpu.3.memoryAllocatedBytes": 80246210560.0, "system.gpu.0.memory": 46.93, "system.gpu.process.0.memory": 46.93, "system.gpu.1.memory": 54.8, "system.gpu.2.memory": 54.13, "system.gpu.3.memory": 48.33, "system.gpu.0.gpu": 99.87, "system.gpu.process.0.gpu": 99.87, "system.gpu.1.gpu": 99.93, "system.gpu.2.gpu": 99.93, "system.gpu.3.gpu": 99.93, "system.gpu.0.temp": 49.27, "system.gpu.process.0.temp": 49.27, "system.gpu.1.temp": 44.33, "system.gpu.2.temp": 52.33, "system.gpu.3.temp": 45.27, "system.gpu.0.powerWatts": 318.96, "system.gpu.process.0.powerWatts": 318.96, "system.gpu.1.powerWatts": 321.14, "system.gpu.2.powerWatts": 316.9, "system.gpu.3.powerWatts": 294.91, "system.gpu.0.powerPercent": 63.79, "system.gpu.process.0.powerPercent": 63.79, "system.gpu.1.powerPercent": 64.23, "system.gpu.2.powerPercent": 63.38, "system.gpu.3.powerPercent": 58.98, "_wandb": true, "_timestamp": 1758161030.272879, "_runtime": 14732.681452}']}, 'output.log': {'offset': 328, 'content': ['ERROR 2025-09-18T02:03:48.991299 Training progress : 8%|â–Š | 140/1687 [03:16<35:44, 1.39s/it]\n']}}, 'dropped': 0}} 2025-09-18 02:03:59,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:01,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:01,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:03,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:03,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:05,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:06,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:04:07,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:08,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:09,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:11,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:11,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:13,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:14,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:15,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:16,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:17,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:19,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:19,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:20,279 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:04:21,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:04:21,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:23,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:25,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:25,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:26,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:27,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:29,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:30,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:31,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:31,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:33,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:35,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:36,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:04:36,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:37,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:39,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:41,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:41,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:43,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:45,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:46,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:47,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:47,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:49,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:50,282 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:04:51,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:04:51,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:52,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:53,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:55,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:56,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:04:57,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:04:58,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:04:59,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:01,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:01,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:03,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:04,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:05,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:06,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:05:07,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:09,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:09,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:11,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:11,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:13,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:15,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:15,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:16,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:17,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:19,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:20,287 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:05:20,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:21,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:05:21,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:23,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:25,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:26,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:26,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:27,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:29,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:31,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:31,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:31,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:33,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:35,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:36,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:05:37,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:37,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:39,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:41,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:41,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:42,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:43,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:45,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:46,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:47,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:48,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:49,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:50,287 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:05:51,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:05:51,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:53,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:53,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:55,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:56,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:05:57,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:05:59,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:05:59,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:01,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:01,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:03,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:04,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:05,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:06,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:06:07,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:09,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:10,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:11,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:11,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:13,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:15,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:16,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:16,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:17,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:19,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:20,290 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:06:21,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:06:21,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:21,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:23,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:25,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:26,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:27,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:27,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:29,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:31,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:31,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:32,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:33,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:35,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:36,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:06:37,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:38,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:39,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:41,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:41,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:43,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:43,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:45,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:46,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:47,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:49,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:49,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:50,292 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:06:51,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:06:51,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:53,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:54,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:06:55,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:56,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:06:57,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:06:59,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:00,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:01,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:01,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:03,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:05,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:05,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:06,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:07:07,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:09,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:11,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:11,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:11,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:13,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:15,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:16,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:16,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:17,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:19,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:20,296 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:07:21,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:07:21,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:22,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:23,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:25,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:26,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:27,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:27,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:29,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:31,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:31,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:33,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:33,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:35,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:36,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:07:37,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:39,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:39,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:41,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:41,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:43,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:44,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:45,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:46,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:47,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:49,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:50,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:50,298 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:07:51,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:07:51,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:53,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:55,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:07:55,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:56,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:07:57,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:07:59,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:01,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:01,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:01,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:03,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:05,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:06,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:08:06,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:07,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:09,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:11,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:11,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:12,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:13,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:15,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:16,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:17,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:17,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:19,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:20,301 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:08:21,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:08:21,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:23,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:23,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:25,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:26,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:27,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:28,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:29,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:31,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:31,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:33,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:34,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:35,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:36,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:08:37,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:39,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:39,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:41,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:41,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:43,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:45,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:45,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:46,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:47,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:49,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:50,304 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:08:51,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:08:51,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:51,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:53,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:55,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:56,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:08:56,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:08:57,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:08:59,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:01,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:01,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:02,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:03,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:05,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:06,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:09:07,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:07,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:09,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:11,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:11,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:13,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:13,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:15,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:16,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:17,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:18,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:19,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:20,307 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:09:21,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:09:21,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:23,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:24,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:25,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:26,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:27,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:29,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:29,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:31,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:31,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:33,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:35,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:35,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:36,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:09:37,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:39,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:40,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:41,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:41,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:43,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:45,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:46,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:46,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:47,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:49,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:50,309 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:09:51,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:09:51,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:51,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:53,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:55,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:56,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:09:57,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:09:57,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:09:59,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:01,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:01,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:03,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:03,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:05,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:06,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:10:07,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:08,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:09,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:11,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:11,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:13,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:14,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:15,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:16,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:17,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:19,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:19,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:20,313 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:10:21,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:10:21,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:23,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:25,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:25,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:26,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:27,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:29,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:30,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:31,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:31,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:33,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:35,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:36,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:10:36,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:37,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:39,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:41,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:41,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:41,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:43,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:45,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:46,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:47,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:47,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:49,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:50,317 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:10:51,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:10:51,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:52,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:53,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:55,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:56,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:10:57,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:10:58,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:10:59,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:01,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:01,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:03,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:03,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:05,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:06,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:11:07,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:09,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:09,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:11,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:11,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:13,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:15,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:15,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:16,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:17,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:20,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:20,319 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:11:20,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:21,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:11:22,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:24,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:26,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:26,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:26,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:28,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:30,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:31,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:31,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:32,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:34,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:36,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:36,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:11:37,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:38,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:40,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:41,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:42,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:42,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:44,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:46,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:46,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:48,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:48,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:50,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:50,321 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:11:51,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:11:52,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:53,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:11:54,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:56,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:56,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:11:58,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:11:59,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:00,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:01,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:02,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:04,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:04,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:06,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:06,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:12:08,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:10,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:10,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:11,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:12,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:14,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:15,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:16,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:16,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:18,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:20,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:20,324 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:12:21,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:12:21,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:22,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:24,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:26,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:26,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:26,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:28,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:30,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:31,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:32,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:32,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:34,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:36,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:36,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:12:38,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:38,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:40,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:41,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:42,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:43,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:44,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:46,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:46,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:48,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:49,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:50,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:50,328 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:12:51,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:12:52,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:54,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:54,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:12:56,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:12:56,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:12:58,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:00,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:00,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:01,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:02,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:04,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:05,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:06,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:13:06,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:08,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:10,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:11,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:11,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:12,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:14,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:16,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:16,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:16,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:18,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:20,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:20,330 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:13:21,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:13:22,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:22,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:24,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:26,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:26,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:27,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:28,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:30,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:31,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:32,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:33,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:34,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:36,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:13:36,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:38,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:38,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:40,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:41,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:42,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:44,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:44,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:46,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:46,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:48,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:50,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:50,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:50,333 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:13:51,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:13:52,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:54,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:55,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:13:56,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:13:56,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:13:58,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:00,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:01,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:01,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:02,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:04,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:06,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:14:06,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:06,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:08,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:10,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:11,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:12,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:12,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:14,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:16,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:16,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:17,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:18,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:20,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:20,336 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:14:21,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:14:22,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:23,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:24,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:26,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:26,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:28,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:28,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:30,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:31,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:32,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:34,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:34,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:36,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:14:36,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:38,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:39,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:40,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:41,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:42,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:44,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:45,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:46,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:46,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:48,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:50,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:50,341 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:14:51,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:14:51,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:52,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:54,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:56,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:14:56,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:14:56,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:14:58,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:00,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:01,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:02,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:02,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:04,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:06,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:15:06,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:07,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:08,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:10,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:11,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:12,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:13,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:14,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:16,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:16,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:18,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:18,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:20,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:20,341 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:15:21,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:15:22,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:24,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:24,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:26,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:26,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:28,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:29,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:30,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:31,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:32,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:34,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:35,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:36,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:15:36,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:38,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:40,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:40,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:41,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:42,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:44,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:46,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:46,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:46,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:48,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:50,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:50,344 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:15:51,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:15:51,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:52,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:54,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:56,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:15:56,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:15:57,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:15:58,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:00,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:01,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:02,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:03,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:04,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:06,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:16:06,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:08,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:08,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:10,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:11,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:12,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:14,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:14,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:16,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:16,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:18,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:19,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:20,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:20,347 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:16:21,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:16:22,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:24,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:25,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:26,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:26,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:28,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:30,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:30,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:31,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:32,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:34,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:36,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:16:36,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:36,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:38,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:40,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:41,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:41,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:42,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:44,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:46,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:46,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:47,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:48,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:50,351 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:16:50,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:51,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:16:52,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:52,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:54,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:56,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:16:56,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:16:58,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:16:58,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:00,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:01,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:02,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:03,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:04,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:06,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:17:06,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:08,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:09,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:10,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:11,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:12,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:14,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:14,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:16,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:16,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:18,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:20,352 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:17:20,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:20,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:21,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:17:22,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:24,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:26,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:26,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:26,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:28,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:30,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:31,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:31,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:32,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:34,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:36,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:17:36,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:37,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:38,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:40,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:41,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:42,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:42,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:44,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:46,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:46,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:48,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:48,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:50,355 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:17:50,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:51,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:17:52,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:53,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:17:54,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:56,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:56,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:17:58,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:17:59,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:00,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:01,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:02,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:04,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:04,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:06,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:18:06,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:08,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:10,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:10,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:11,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:12,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:14,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:15,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:16,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:16,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:18,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:20,358 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:18:20,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:21,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:18:21,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:22,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:24,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:26,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:26,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:26,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:28,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:30,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:31,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:32,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:32,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:34,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:36,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:18:36,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:38,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:38,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:40,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:41,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:42,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:43,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:44,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:46,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:46,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:48,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:49,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:50,360 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:18:50,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:51,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:18:52,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:54,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:54,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:18:56,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:18:56,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:18:58,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:00,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:00,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:01,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:02,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:04,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:05,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:06,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:19:06,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:08,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:10,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:11,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:11,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:12,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:14,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:16,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:16,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:16,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:18,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:20,363 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:19:20,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:21,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:19:22,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:22,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:24,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:26,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:26,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:27,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:28,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:30,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:31,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:32,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:33,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:34,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:36,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:19:36,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:38,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:38,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:40,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:41,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:42,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:44,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:44,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:46,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:46,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:48,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:49,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:50,366 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:19:50,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:51,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:19:52,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:54,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:55,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:19:56,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:19:56,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:19:58,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:00,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:01,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:01,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:02,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:04,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:06,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:20:06,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:06,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:08,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:10,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:11,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:12,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:12,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:14,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:16,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:16,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:17,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:18,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:20,369 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:20:20,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:21,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:20:22,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:23,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:24,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:26,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:26,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:28,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:28,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:30,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:31,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:32,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:34,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:34,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:36,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:20:36,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:38,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:39,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:40,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:41,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:42,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:44,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:45,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:46,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:46,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:48,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:50,372 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:20:50,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:50,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:51,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:20:51,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:51,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:20:52,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:54,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:55,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:20:56,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:20:56,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:20:58,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:00,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:00,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:01,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:02,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:04,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:06,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:21:06,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:06,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:08,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:10,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:11,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:11,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:12,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:14,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:16,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:16,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:17,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:18,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:20,375 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:21:20,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:21,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:21:22,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:22,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:24,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:26,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:26,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:28,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:28,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:30,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:31,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:32,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:33,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:34,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:36,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:21:36,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:38,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:39,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:40,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:41,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:42,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:44,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:45,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:46,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:46,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:48,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:50,378 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:21:50,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:50,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:51,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:21:52,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:54,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:56,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:21:56,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:21:56,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:21:58,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:00,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:01,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:01,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:02,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:04,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:06,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:22:06,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:07,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:08,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:10,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:11,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:12,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:12,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:14,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:16,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:16,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:18,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:18,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:20,383 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:22:20,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:21,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:22:22,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:23,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:24,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:26,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:26,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:28,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:29,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:30,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:31,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:32,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:34,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:34,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:36,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:22:36,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:38,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:40,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:40,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:41,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:42,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:44,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:45,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:46,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:46,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:48,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:50,384 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:22:50,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:51,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:22:51,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:52,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:54,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:56,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:22:56,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:22:57,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:22:58,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:00,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:01,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:02,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:02,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:04,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:06,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:23:06,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:08,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:08,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:10,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:11,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:12,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:13,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:14,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:16,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:16,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:18,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:19,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:20,386 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:23:20,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:21,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:23:22,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:24,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:24,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:26,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:26,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:28,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:30,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:30,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:31,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:32,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:34,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:35,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:36,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:23:36,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:38,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:40,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:41,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:41,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:42,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:44,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:46,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:46,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:46,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:48,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:50,389 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:23:50,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:51,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:23:52,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:52,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:54,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:56,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:23:56,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:23:57,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:23:58,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:00,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:01,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:02,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:03,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:04,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:06,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:24:06,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:08,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:08,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:10,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:11,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:12,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:14,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:14,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:16,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:16,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:18,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:20,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:20,393 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:24:20,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:21,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:24:22,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:24,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:25,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:26,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:26,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:28,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:30,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:31,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:31,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:32,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:34,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:36,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:24:36,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:36,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:38,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:40,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:41,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:42,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:42,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:44,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:46,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:46,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:47,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:48,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:50,394 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:24:50,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:51,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:24:52,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:53,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:54,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:56,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:24:56,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:24:58,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:24:58,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:00,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:01,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:02,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:04,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:04,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:06,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:25:06,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:08,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:09,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:10,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:11,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:12,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:14,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:15,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:16,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:16,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:18,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:20,397 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:25:20,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:20,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:21,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:25:22,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:24,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:25,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:26,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:26,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:28,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:30,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:30,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:31,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:32,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:34,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:36,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:25:36,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:36,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:38,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:40,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:41,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:41,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:42,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:44,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:46,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:46,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:47,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:48,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:50,400 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:25:50,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:51,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:25:52,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:52,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:54,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:56,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:25:56,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:25:58,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:25:58,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:00,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:01,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:02,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:03,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:04,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:06,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:26:06,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:08,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:09,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:10,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:11,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:12,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:14,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:14,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:16,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:16,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:18,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:20,405 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:26:20,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:20,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:21,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:26:22,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:24,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:26,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:26,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:26,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:28,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:30,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:31,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:31,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:32,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:34,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:36,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:26:36,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:37,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:38,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:40,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:41,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:42,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:42,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:44,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:46,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:46,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:48,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:48,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:50,406 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:26:50,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:51,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:26:52,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:53,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:26:54,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:56,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:26:56,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:58,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:26:59,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:00,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:01,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:02,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:04,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:04,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:06,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:27:06,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:08,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:10,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:10,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:11,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:12,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:14,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:15,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:16,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:16,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:18,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:20,409 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:27:20,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:21,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:27:21,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:22,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:24,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:26,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:26,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:26,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:28,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:30,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:31,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:32,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:32,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:34,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:36,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:27:36,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:38,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:38,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:41,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:41,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:43,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:43,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:45,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:46,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:47,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:49,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:49,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:50,411 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:27:51,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:51,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:27:53,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:54,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:27:55,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:56,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:27:57,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:27:59,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:00,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:01,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:01,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:03,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:05,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:05,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:06,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:28:07,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:09,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:11,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:11,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:11,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:13,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:15,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:16,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:16,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:17,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:19,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:20,414 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:28:21,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:21,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:28:22,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:23,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:25,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:26,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:27,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:27,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:29,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:31,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:31,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:33,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:33,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:35,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:36,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:28:37,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:38,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:39,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:41,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:41,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:43,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:44,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:45,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:46,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:47,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:49,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:49,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:50,417 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:28:51,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:51,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:28:53,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:55,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:55,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:28:56,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:28:57,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:28:59,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:01,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:01,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:01,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:03,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:05,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:06,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:29:06,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:07,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:09,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:11,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:11,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:12,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:13,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:15,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:16,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:17,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:17,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:19,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:20,421 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:29:21,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:29:21,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:23,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:23,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:25,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:26,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:27,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:28,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:29,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:31,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:31,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:33,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:34,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:35,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:36,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:29:37,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:39,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:39,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:41,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:41,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:43,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:45,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:45,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:46,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:47,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:49,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:50,424 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:29:50,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:51,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:29:51,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:53,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:55,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:55,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:29:56,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:29:57,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:29:59,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:00,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:01,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:01,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:03,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:05,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:06,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:30:06,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:07,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:09,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:11,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:11,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:11,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:13,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:15,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:16,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:17,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:17,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:19,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:20,426 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:30:21,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:30:21,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:22,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:23,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:25,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:26,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:27,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:28,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:29,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:31,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:31,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:33,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:33,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:35,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:36,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:30:37,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:39,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:39,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:41,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:41,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:43,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:44,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:45,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:46,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:47,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:49,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:50,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:50,430 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:30:51,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:30:51,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:53,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:55,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:55,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:30:56,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:30:57,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:30:59,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:01,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:01,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:01,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:03,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:05,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:06,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:31:06,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:07,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:09,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:11,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:11,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:12,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:13,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:15,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:16,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:17,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:18,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:19,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:20,433 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:31:21,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:31:21,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:23,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:23,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:25,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:26,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:27,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:29,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:29,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:31,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:31,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:33,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:34,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:35,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:36,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:31:37,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:39,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:40,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:41,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:41,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:43,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:45,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:45,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:46,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:47,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:49,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:50,437 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:31:51,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:31:51,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:51,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:53,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:55,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:56,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:31:56,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:31:57,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:31:59,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:01,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:01,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:02,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:03,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:05,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:06,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:32:07,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:07,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:09,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:11,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:11,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:13,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:13,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:15,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:16,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:17,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:18,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:19,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:20,439 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:32:21,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:32:21,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:23,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:24,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:25,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:26,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:27,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:29,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:30,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:31,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:31,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:33,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:35,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:35,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:36,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:32:37,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:39,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:41,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:41,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:41,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:43,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:45,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:46,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:46,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:47,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:49,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:50,442 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:32:51,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:32:51,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:52,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:53,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:55,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:56,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:32:57,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:32:57,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:32:59,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:01,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:01,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:03,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:03,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:05,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:06,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:33:07,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:08,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:09,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:11,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:11,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:13,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:14,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:15,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:16,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:17,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:19,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:19,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:20,444 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:33:21,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:33:21,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:23,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:25,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:25,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:26,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:27,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:29,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:30,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:31,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:31,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:33,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:35,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:36,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:33:36,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:37,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:39,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:41,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:41,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:41,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:43,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:45,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:46,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:47,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:47,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:49,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:50,451 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:33:51,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:33:51,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:53,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:53,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:55,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:56,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:33:57,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:33:58,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:33:59,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:01,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:01,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:03,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:04,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:05,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:06,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:34:07,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:09,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:09,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:11,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:11,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:13,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:15,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:15,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:16,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:17,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:19,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:20,452 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:34:20,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:21,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:34:21,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:23,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:25,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:26,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:26,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:27,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:29,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:31,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:31,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:31,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:33,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:35,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:36,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:34:37,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:37,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:39,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:41,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:41,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:42,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:43,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:45,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:46,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:47,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:48,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:49,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:50,454 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:34:51,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:34:51,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:53,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:53,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:34:55,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:56,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:34:57,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:59,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:34:59,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:01,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:01,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:03,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:05,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:05,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:06,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:35:07,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:09,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:10,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:11,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:11,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:13,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:15,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:16,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:16,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:17,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:19,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:20,457 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:35:21,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:35:21,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:21,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:23,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:25,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:26,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:27,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:27,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:29,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:31,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:31,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:32,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:33,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:35,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:36,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:35:37,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:38,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:39,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:41,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:41,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:43,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:43,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:45,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:46,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:47,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:49,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:49,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:50,460 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:35:51,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:35:51,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:53,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:54,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:35:55,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:56,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:35:57,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:35:59,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:00,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:01,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:01,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:03,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:05,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:05,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:06,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:36:07,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:09,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:11,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:11,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:11,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:13,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:15,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:16,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:16,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:17,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:19,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:20,464 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:36:21,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:36:21,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:22,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:23,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:25,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:26,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:27,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:28,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:29,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:31,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:31,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:33,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:33,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:35,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:36,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:36:37,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:39,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:39,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:41,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:41,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:43,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:44,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:45,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:46,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:47,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:49,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:50,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:50,467 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:36:51,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:36:51,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:53,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:55,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:55,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:36:56,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:36:57,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:36:59,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:01,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:01,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:01,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:03,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:05,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:06,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:37:06,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:07,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:09,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:11,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:11,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:12,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:13,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:15,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:16,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:17,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:17,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:19,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:20,472 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:37:21,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:37:21,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:23,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:23,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:25,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:26,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:27,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:28,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:29,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:31,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:31,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:33,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:34,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:35,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:36,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:37:37,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:39,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:40,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:41,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:41,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:43,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:45,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:45,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:46,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:47,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:49,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:50,503 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:37:51,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:37:51,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:51,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:53,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:55,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:56,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:37:56,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:37:57,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:37:59,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:01,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:01,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:02,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:03,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:05,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:06,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:38:07,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:07,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:09,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:11,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:11,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:13,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:13,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:15,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:16,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:17,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:18,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:19,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:20,506 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:38:21,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:38:21,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:23,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:24,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:25,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:26,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:27,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:29,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:29,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:31,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:31,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:33,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:35,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:35,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:36,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:38:37,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:39,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:40,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:41,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:41,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:43,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:45,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:46,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:46,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:47,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:49,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:50,509 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:38:51,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:38:51,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:52,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:53,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:55,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:56,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:38:57,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:38:57,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:38:59,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:01,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:01,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:03,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:03,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:05,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:06,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:39:07,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:08,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:09,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:11,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:11,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:13,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:14,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:15,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:16,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:17,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:19,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:19,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:20,514 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:39:21,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:39:21,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:23,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:25,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:25,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:26,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:27,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:29,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:30,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:31,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:31,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:33,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:35,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:35,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:36,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:39:37,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:39,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:40,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:41,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:41,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:43,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:45,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:46,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:46,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:47,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:49,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:50,515 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:39:51,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:39:51,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:51,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:53,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:55,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:56,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:39:56,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:39:57,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:39:59,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:01,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:01,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:02,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:03,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:05,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:06,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:40:07,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:07,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:09,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:11,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:11,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:12,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:40:12,541 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 02:40:12,549 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 02:40:12,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:12,560 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 02:40:12,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 02:40:13,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:17,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:17,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:20,518 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:40:21,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:40:23,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:27,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:28,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:28,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:33,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:33,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:40:36,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:40:38,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:40:38,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:40:38,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 02:40:38,394 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 02:40:38,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:40:38,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_7_15593048a2d6b7646c62.table.json 2025-09-18 02:40:38,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 02:40:39,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:40:43,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:48,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:40:51,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:40:56,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:02,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:06,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:41:11,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:17,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:21,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:41:26,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:32,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:36,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:41:41,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:47,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:41:51,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:41:56,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:02,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:06,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:42:11,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:17,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:21,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:42:26,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:32,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:36,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:42:41,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:47,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:49,459 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpl0iqiwsu 2025-09-18 02:42:49,515 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/xvv9vs1h-media/table/rollouts_7_15593048a2d6b7646c62.table.json 2025-09-18 02:42:50,110 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MTk4MDMwNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTg1MDQ3OA==', 'versionIndex': 5}}} 2025-09-18 02:42:50,111 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 02:42:50,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:42:50,111 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_7_15593048a2d6b7646c62.table.json with policy now 2025-09-18 02:42:50,112 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:42:50,113 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:42:50,115 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:42:50,116 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:42:50,471 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/0kcmts2g-media/table/rollouts_7_15593048a2d6b7646c62.table.json 2025-09-18 02:42:50,540 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:42:51,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:42:51,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:42:53,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:42:55,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:42:55,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:42:56,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:42:57,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:42:59,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:01,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:01,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:02,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:03,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:05,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:06,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:43:07,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:07,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:09,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:11,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:11,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:12,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:13,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:15,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:17,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:17,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:18,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:19,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:20,540 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:43:21,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:43:21,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:23,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:23,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:25,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:26,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:27,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:29,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:29,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:31,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:32,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:33,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:34,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:35,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:36,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:43:37,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:39,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:40,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:41,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:41,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:43,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:45,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:45,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:47,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:47,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:49,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:50,544 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:43:51,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:43:51,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:51,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:53,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:55,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:56,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:43:56,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:43:57,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:43:59,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:01,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:02,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:02,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:03,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:05,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:06,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:44:07,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:07,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:10,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:11,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:12,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:13,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:14,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:16,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:17,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:18,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:19,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:20,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:20,545 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:44:21,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:44:22,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:24,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:24,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:26,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:26,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:28,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:30,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:30,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:32,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:32,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:34,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:35,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:36,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:36,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:44:38,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:40,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:41,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:41,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:42,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:44,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:46,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:46,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:47,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:48,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:50,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:50,548 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:44:51,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:44:52,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:52,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:54,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:56,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:44:56,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:44:57,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:44:58,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:00,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:02,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:02,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:03,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:03,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,746 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:03,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:04,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:04,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:04,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:04,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 02:45:06,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:06,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:45:08,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:08,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:10,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:11,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:12,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:14,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:14,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:16,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:17,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:18,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:20,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:20,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:20,551 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:45:21,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:45:22,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:24,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:25,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:26,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:26,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:28,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:30,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:31,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:32,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:32,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:34,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:36,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:36,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:45:36,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:38,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:40,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:41,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:42,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:42,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:44,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:46,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:47,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:47,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:48,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:50,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:50,561 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:45:51,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:45:52,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:53,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:45:54,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:56,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:56,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:45:58,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:45:58,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:00,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:01,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:02,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:04,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:04,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:06,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:06,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:46:08,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:09,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:10,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:11,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:12,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:14,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:15,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:16,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:16,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:18,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:20,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:20,556 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:46:20,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:21,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:46:22,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:24,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:26,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:26,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:26,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:28,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:30,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:31,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:32,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:32,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:34,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:36,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:46:36,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:37,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:38,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:40,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:41,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:42,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:43,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:44,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:46,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:46,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:48,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:48,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:50,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:50,559 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:46:51,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:46:52,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:54,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:54,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:46:56,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:46:56,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:58,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:46:59,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:00,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:01,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:02,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:04,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:05,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:06,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:47:06,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:08,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:10,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:10,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:11,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:12,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:14,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:16,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:16,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:16,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:18,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:20,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:20,562 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:47:21,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:47:21,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:22,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:24,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:26,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:26,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:27,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:28,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:30,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:31,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:32,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:32,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:34,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:36,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:47:36,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:38,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:38,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:40,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:41,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:42,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:44,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:44,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:46,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:46,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:48,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:49,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:50,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:50,565 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:47:51,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:47:52,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:54,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:55,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:47:56,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:47:56,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:47:58,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:00,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:00,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:01,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:02,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:04,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:06,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:48:06,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:06,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:08,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:10,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:11,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:11,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:12,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:14,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:16,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:16,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:17,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:18,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:20,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:20,569 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:48:21,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:48:22,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:22,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:24,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:26,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:26,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:28,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:28,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:30,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:31,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:32,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:33,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:34,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:36,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:48:36,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:38,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:39,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:40,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:41,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:42,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:44,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:44,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:46,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:46,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:48,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:50,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:50,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:50,572 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:48:51,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:48:52,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:54,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:56,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:48:56,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:48:56,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:48:58,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:00,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:01,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:01,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:02,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:04,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:06,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:49:06,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:07,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:08,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:10,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:11,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:12,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:12,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:14,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:16,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:16,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:18,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:18,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:20,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:20,574 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:49:21,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:49:22,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:23,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:24,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:26,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:26,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:28,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:29,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:30,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:31,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:32,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:34,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:34,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:36,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:49:36,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:38,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:40,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:40,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:41,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:42,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:44,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:45,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:46,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:46,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:48,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:50,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:50,577 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:49:51,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:49:51,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:52,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:54,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:56,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:49:56,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:49:56,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:49:58,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:00,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:01,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:02,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:02,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:04,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:06,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:50:06,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:08,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:08,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:10,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:11,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:12,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:13,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:14,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:16,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:16,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:18,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:19,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:20,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:20,580 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:50:21,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:50:22,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:24,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:24,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:26,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:26,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:28,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:30,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:30,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:31,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:32,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:34,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:35,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:36,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:50:36,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:38,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:40,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:41,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:41,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:42,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:44,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:46,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:46,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:46,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:48,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:50,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:50,582 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:50:51,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:50:52,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:52,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:54,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:56,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:50:56,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:50:57,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:50:58,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:00,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:01,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:02,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:03,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:04,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:06,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:51:06,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:08,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:09,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:10,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:11,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:12,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:14,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:14,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:16,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:16,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:18,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:20,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:20,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:20,588 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:51:21,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:51:22,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:24,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:25,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:26,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:26,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:28,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:30,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:31,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:31,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:32,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:34,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:36,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:51:36,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:36,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:38,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:40,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:41,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:42,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:42,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:44,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:46,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:46,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:47,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:48,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:50,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:50,588 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:51:51,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:51:52,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:53,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:51:54,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:56,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:51:56,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:58,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:51:58,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:00,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:01,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:02,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:04,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:04,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:06,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:52:06,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:08,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:09,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:10,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:11,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:12,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:14,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:15,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:16,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:16,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:18,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:20,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:20,592 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:52:20,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:21,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:52:22,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:24,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:26,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:26,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:26,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:28,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:30,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:31,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:32,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:32,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:34,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:36,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:52:36,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:37,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:38,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:40,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:41,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:42,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:43,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:44,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:46,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:46,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:48,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:48,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:50,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:50,595 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:52:51,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:52:52,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:54,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:52:54,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:56,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:52:56,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:58,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:52:59,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:00,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:01,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:02,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:04,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:05,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:06,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:53:06,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:08,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:10,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:10,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:11,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:12,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:14,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:16,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:16,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:16,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:18,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:20,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:20,599 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:53:21,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:53:21,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:22,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:24,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:26,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:26,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:27,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:28,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:30,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:31,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:32,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:33,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:34,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:36,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:53:36,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:38,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:38,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:40,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:41,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:42,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:44,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:44,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:46,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:46,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:48,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:49,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:50,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:50,600 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:53:51,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:53:52,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:54,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:55,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:53:56,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:53:56,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:53:58,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:00,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:00,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:01,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:02,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:04,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:06,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:54:06,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:06,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:08,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:10,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:11,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:11,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:12,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:14,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:16,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:16,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:17,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:18,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:20,603 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:54:20,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:21,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:54:22,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:22,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:24,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:26,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:26,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:28,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:28,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:30,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:31,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:32,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:33,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:34,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:36,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:54:36,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:38,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:39,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:40,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:41,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:42,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:44,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:45,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:46,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:46,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:48,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:50,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:50,607 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:54:50,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:51,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:54:52,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:54,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:56,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:54:56,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:54:56,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:54:58,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:00,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:01,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:01,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:02,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:04,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:06,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:55:06,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:07,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:08,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:10,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:11,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:12,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:12,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:14,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:16,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:16,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:18,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:18,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:20,611 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:55:20,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:21,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:55:22,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:23,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:24,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:26,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:26,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:28,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:29,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:30,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:31,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:32,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:34,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:34,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:36,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:55:36,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:38,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:40,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:40,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:41,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:42,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:44,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:45,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:46,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:46,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:48,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:50,613 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:55:50,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:51,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:55:51,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:52,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:54,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:56,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:55:56,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:55:57,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:55:58,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:00,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:01,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:02,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:02,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:04,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:06,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:56:06,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:08,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:08,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:10,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:11,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:12,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:13,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:14,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:16,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:16,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:18,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:19,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:20,615 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:56:20,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:21,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:56:22,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:24,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:24,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:26,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:26,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:28,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:30,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:30,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:31,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:32,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:34,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:35,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:36,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:56:36,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:38,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:40,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:41,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:41,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:42,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:44,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:46,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:46,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:46,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:48,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:50,617 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:56:50,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:51,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:56:52,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:52,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:54,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:56,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:56:56,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:56:57,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:56:58,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:00,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:01,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:02,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:03,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:04,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:06,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:57:06,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:08,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:08,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:10,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:11,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:12,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:14,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:14,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:16,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:16,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:18,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:20,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:20,623 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:57:20,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:21,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:57:22,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:24,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:25,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:26,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:26,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:28,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:30,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:31,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:31,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:32,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:34,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:36,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:57:36,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:36,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:38,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:40,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:41,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:42,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:42,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:44,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:46,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:46,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:47,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:48,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:50,623 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:57:50,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:51,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:57:52,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:53,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:57:54,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:56,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:57:56,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:58,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:57:58,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:00,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:01,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:02,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:04,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:04,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:06,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:58:06,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:08,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:09,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:10,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:11,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:12,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:14,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:15,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:16,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:16,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:18,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:20,626 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:58:20,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:20,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:21,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:58:22,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:24,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:26,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:26,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:26,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:28,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:30,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:31,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:32,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:32,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:34,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:36,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:58:36,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:37,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:38,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:40,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:41,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:42,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:43,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:44,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:46,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:46,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:48,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:48,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:50,629 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:58:50,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:51,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:58:52,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:54,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:58:54,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:56,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:58:56,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:58,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:58:59,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:00,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:01,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:02,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:04,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:05,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:06,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:59:06,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:08,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:10,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:10,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:11,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:12,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:14,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:16,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:16,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:16,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:18,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:20,631 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:59:20,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:21,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:59:21,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:22,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:24,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:26,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:26,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:27,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:28,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:30,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:31,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:32,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:32,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:34,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:36,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:59:36,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:38,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:38,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:40,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:41,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:42,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:44,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:44,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:46,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:46,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:48,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:49,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:50,634 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 02:59:50,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:51,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 02:59:52,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:54,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:55,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 02:59:56,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 02:59:56,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 02:59:58,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:00,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:00,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:01,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:02,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:04,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:06,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:00:06,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:06,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:08,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:10,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:11,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:11,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:12,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:14,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:16,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:16,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:17,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:18,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:20,637 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:00:20,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:21,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:00:22,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:22,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:24,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:26,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:26,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:28,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:28,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:30,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:31,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:32,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:33,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:34,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:36,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:00:36,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:38,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:39,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:40,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:41,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:42,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:44,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:44,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:46,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:46,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:48,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:50,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:50,639 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:00:50,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:51,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:00:52,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:54,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:56,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:00:56,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:00:56,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:00:58,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:01,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:01,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:01,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:03,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:05,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:06,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:01:07,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:07,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:09,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:11,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:11,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:12,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:13,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:15,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:16,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:17,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:18,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:19,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:20,642 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:01:21,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:21,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:01:23,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:23,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:25,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:26,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:27,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:29,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:29,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:31,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:31,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:33,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:34,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:35,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:36,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:01:37,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:39,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:40,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:41,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:41,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:43,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:45,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:45,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:46,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:47,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:49,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:50,645 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:01:51,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:51,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:01:51,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:53,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:55,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:56,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:56,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:01:56,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:01:57,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:01:59,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:01,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:01,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:02,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:03,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:05,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:06,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:02:07,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:07,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:09,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:11,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:11,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:13,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:13,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:15,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:16,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:17,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:19,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:19,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:20,648 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:02:21,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:21,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:02:23,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:24,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:25,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:26,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:27,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:29,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:30,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:31,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:31,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:33,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:35,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:35,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:36,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:02:37,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:39,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:41,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:41,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:41,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:43,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:45,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:46,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:46,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:47,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:49,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:50,653 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:02:51,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:51,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:02:52,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:53,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:55,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:56,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:02:57,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:02:57,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:02:59,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:01,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:01,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:03,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:03,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:05,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:06,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:03:07,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:08,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:09,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:11,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:11,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:13,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:14,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:15,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:16,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:17,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:19,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:20,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:20,653 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:03:21,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:21,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:03:23,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:25,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:25,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:26,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:27,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:29,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:31,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:31,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:31,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:33,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:35,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:36,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:03:36,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:37,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:39,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:41,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:41,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:42,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:43,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:45,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:46,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:47,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:47,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:49,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:50,657 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:03:51,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:03:51,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:53,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:53,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:55,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:56,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:03:57,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:03:58,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:03:59,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:01,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:01,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:03,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:04,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:05,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:06,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:04:07,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:09,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:09,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:11,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:11,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:13,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:15,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:15,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:16,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:17,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:19,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:20,659 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:04:20,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:21,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:04:21,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:23,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:25,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:26,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:26,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:27,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:29,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:31,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:31,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:32,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:33,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:35,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:36,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:04:37,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:37,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:39,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:41,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:41,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:43,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:43,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:45,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:46,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:47,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:48,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:49,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:50,666 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:04:51,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:04:51,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:53,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:54,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:04:55,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:56,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:04:57,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:59,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:04:59,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:01,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:02,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:03,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:05,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:06,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:05:06,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:07,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:09,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:10,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:11,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:11,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:13,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:15,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:16,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:16,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:17,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:19,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:20,664 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:05:21,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:05:21,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:21,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:23,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:25,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:26,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:27,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:27,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:29,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:31,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:31,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:33,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:33,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:35,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:36,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:05:37,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:38,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:39,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:41,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:41,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:43,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:44,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:46,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:46,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:48,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:49,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:50,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:50,667 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:05:51,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:05:52,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:54,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:55,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:05:56,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:05:56,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:05:58,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:00,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:00,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:01,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:02,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:04,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:06,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:06:06,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:06,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:08,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:10,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:11,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:11,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:12,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:14,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:16,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:16,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:17,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:18,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:20,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:20,670 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:06:21,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:06:22,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:22,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:24,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:26,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:26,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:28,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:28,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:30,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:31,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:32,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:33,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:34,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:36,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:06:36,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:38,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:39,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:40,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:41,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:42,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:44,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:44,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:46,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:46,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:48,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:50,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:50,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:50,675 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:06:51,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:06:52,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:54,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:56,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:06:56,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:06:56,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:06:58,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:00,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:01,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:01,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:02,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:04,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:06,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:07:06,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:07,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:08,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:10,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:11,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:12,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:12,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:14,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:16,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:16,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:18,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:18,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:20,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:20,675 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:07:21,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:07:22,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:23,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:24,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:26,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:26,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:28,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:29,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:30,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:31,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:32,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:34,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:34,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:36,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:07:36,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:38,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:40,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:40,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:41,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:42,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:44,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:45,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:46,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:46,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:48,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:50,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:50,678 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:07:51,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:07:51,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:52,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:54,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:56,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:07:56,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:07:56,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:07:58,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:00,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:01,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:02,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:02,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:04,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:06,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:08:06,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:08,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:08,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:08,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:08,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:08:10,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:11,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:12,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:13,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:14,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:16,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:16,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:18,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:19,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:20,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:20,681 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:08:21,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:08:22,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:24,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:24,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:26,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:26,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:28,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:30,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:30,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:31,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:32,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:34,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:35,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:36,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:08:36,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:38,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:40,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:41,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:41,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:42,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:44,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:46,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:46,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:46,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:48,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:50,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:50,683 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:08:51,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:08:52,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:52,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:54,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:56,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:08:56,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:08:58,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:08:58,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:00,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:01,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:02,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:03,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:04,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:06,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:09:06,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:08,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:09,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:10,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:11,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:12,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:14,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:14,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:16,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:16,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:18,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:20,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:20,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:20,687 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:09:21,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:09:22,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:24,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:25,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:26,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:26,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:28,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:30,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:31,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:31,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:32,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:34,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:36,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:09:36,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:36,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:38,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:40,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:41,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:42,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:42,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:44,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:46,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:46,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:47,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:48,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:50,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:50,690 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:09:51,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:09:52,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:53,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:09:54,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:56,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:09:56,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:58,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:09:58,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:00,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:01,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:02,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:04,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:04,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:06,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:10:06,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:08,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:10,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:10,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:11,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:12,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:14,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:15,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:16,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:16,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:18,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:20,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:20,694 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:10:20,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:21,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:10:22,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:24,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:26,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:26,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:26,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:28,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:30,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:31,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:32,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:32,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:34,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:36,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:10:36,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:37,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:38,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:40,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:41,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:42,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:43,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:44,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:46,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:46,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:48,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:48,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:50,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:50,695 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:10:51,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:10:52,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:54,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:10:54,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:56,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:10:56,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:58,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:10:59,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:00,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:01,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:02,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:04,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:05,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:06,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:11:06,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:08,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:10,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:10,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:11,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:12,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:14,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:16,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:16,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:16,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:18,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:20,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:20,697 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:11:21,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:11:22,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:22,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:24,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:26,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:26,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:27,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:28,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:30,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:31,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:32,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:33,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:34,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:36,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:11:36,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:38,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:38,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:40,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:41,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:42,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:44,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:44,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:46,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:46,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:48,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:49,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:50,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:50,700 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:11:51,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:11:52,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:54,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:55,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:11:56,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:11:56,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:11:58,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:00,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:00,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:01,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:02,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:04,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:06,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:12:06,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:06,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:08,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:10,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:11,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:11,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:12,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:14,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:16,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:16,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:17,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:18,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:20,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:20,707 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:12:21,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:12:22,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:22,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:24,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:26,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:26,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:28,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:28,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:30,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:31,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:32,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:34,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:34,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:36,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:12:36,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:38,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:39,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:40,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:41,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:42,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:44,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:45,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:46,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:46,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:48,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:50,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:50,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:50,706 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:12:51,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:12:52,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:54,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:56,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:12:56,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:12:56,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:12:58,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:00,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:01,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:01,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:02,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:04,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:06,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:13:06,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:07,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:08,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:10,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:11,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:12,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:12,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:14,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:16,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:16,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:18,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:18,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:20,709 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:13:20,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:21,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:13:22,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:23,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:24,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:26,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:26,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:28,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:29,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:30,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:31,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:32,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:34,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:34,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:36,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:13:36,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:38,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:40,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:40,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:41,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:42,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:44,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:45,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:46,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:46,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:48,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:50,713 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:13:50,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:51,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:13:51,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:52,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:54,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:56,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:13:56,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:13:57,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:13:58,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:00,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:01,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:02,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:02,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:04,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:06,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:14:06,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:08,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:08,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:10,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:11,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:12,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:13,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:14,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:16,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:16,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:18,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:19,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:20,718 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:14:20,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:21,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:14:22,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:24,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:24,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:26,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:26,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:28,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:30,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:30,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:31,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:32,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:34,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:35,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:36,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:14:36,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:38,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:40,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:41,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:41,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:42,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:44,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:46,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:46,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:46,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:48,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:50,719 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:14:50,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:51,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:14:52,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:52,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:54,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:56,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:14:56,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:14:58,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:14:58,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:00,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:01,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:02,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:03,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:04,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:06,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:15:06,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:08,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:09,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:10,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:11,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:12,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:14,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:14,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:16,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:16,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:18,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:20,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:20,721 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:15:20,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:21,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:15:22,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:24,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:25,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:26,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:26,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:28,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:30,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:31,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:31,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:32,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:34,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:36,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:15:36,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:36,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:38,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:40,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:41,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:42,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:42,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:44,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:46,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:46,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:47,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:48,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:50,724 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:15:50,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:51,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:15:52,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:53,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:15:54,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:56,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:15:56,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:58,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:15:58,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:00,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:01,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:02,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:04,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:04,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:06,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:16:06,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:08,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:10,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:10,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:11,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:12,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:14,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:15,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:16,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:16,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:18,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:20,727 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:16:20,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:20,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:21,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:16:22,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:24,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:26,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:26,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:26,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:28,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:30,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:31,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:32,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:32,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:34,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:36,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:16:36,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:37,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:38,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:40,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:41,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:42,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:43,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:44,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:46,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:46,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:48,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:48,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:50,730 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:16:50,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:51,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:16:52,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:54,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:16:54,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:56,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:16:56,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:58,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:16:59,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:00,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:01,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:02,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:04,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:05,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:06,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:17:06,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:08,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:10,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:10,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:11,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:12,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:14,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:16,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:16,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:16,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:18,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:20,732 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:17:20,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:21,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:17:22,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:22,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:24,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:26,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:26,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:27,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:28,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:30,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:31,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:32,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:33,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:34,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:36,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:17:36,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:38,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:38,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:40,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:41,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:42,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:44,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:44,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:46,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:46,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:48,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:49,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:50,737 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:17:50,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:51,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:17:52,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:54,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:55,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:17:56,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:17:56,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:17:58,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:00,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:01,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:01,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:03,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:05,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:06,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:18:06,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:07,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:09,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:11,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:11,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:11,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:13,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:15,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:16,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:17,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:17,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:19,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:20,764 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:18:21,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:21,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:18:22,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:23,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:25,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:26,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:27,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:28,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:29,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:31,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:31,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:33,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:34,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:35,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:36,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:18:37,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:39,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:39,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:41,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:41,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:43,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:45,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:45,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:46,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:47,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:49,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:50,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:50,767 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:18:51,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:51,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:18:53,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:55,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:56,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:18:56,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:18:57,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:18:59,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:01,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:01,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:01,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:03,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:05,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:06,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:19:07,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:07,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:09,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:11,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:11,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:12,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:13,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:15,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:16,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:17,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:18,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:19,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:20,770 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:19:21,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:21,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:19:23,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:23,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:25,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:26,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:27,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:29,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:29,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:31,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:31,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:33,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:34,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:35,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:36,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:19:37,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:39,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:40,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:41,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:41,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:43,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:45,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:45,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:46,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:47,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:49,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:50,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:50,774 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:19:51,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:51,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:19:53,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:55,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:55,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:19:56,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:19:57,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:19:59,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:01,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:01,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:01,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:03,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:05,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:06,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:20:06,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:07,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:09,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:11,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:11,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:11,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:13,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:15,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:16,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:16,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:17,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:18,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:20:18,574 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 03:20:18,581 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 03:20:18,591 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 03:20:19,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 03:20:20,778 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:20:21,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:22,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:22,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:20:22,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:27,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:28,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:33,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:34,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:35,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:37,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:20:38,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:42,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:43,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:46,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:20:46,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:20:46,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 03:20:46,938 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 03:20:46,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:20:47,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_8_9b215a1de50b7a925101.table.json 2025-09-18 03:20:47,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 03:20:47,652 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/d3gi3vc6-media/table/rollouts_8_9b215a1de50b7a925101.table.json 2025-09-18 03:20:47,840 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmp5i83atss 2025-09-18 03:20:48,535 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MjExMDU5MA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MTk4MDMwNA==', 'versionIndex': 6}}} 2025-09-18 03:20:48,535 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 03:20:48,535 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_8_9b215a1de50b7a925101.table.json with policy now 2025-09-18 03:20:48,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:48,837 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/3dc8z67b-media/table/rollouts_8_9b215a1de50b7a925101.table.json 2025-09-18 03:20:49,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:49,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:50,780 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:20:51,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:51,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:20:53,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:54,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:20:55,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:57,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:20:57,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:59,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:20:59,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:01,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:02,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:03,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:05,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:05,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:06,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:21:07,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:09,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:10,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:11,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:12,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:13,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:15,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:16,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:17,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:17,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:19,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:20,782 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:21:21,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:21,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:21:22,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:23,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:25,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:27,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:27,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:27,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:29,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:31,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:32,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:33,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:33,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:35,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:36,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:21:37,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:38,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:39,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:41,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:42,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:43,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:44,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:45,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:47,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:47,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:49,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:49,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:50,785 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:21:51,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:51,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:21:53,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:55,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:55,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:21:57,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:21:57,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:21:59,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:00,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:01,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:02,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:03,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:05,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:06,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:06,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:22:07,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:09,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:11,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:11,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:12,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:13,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:15,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:17,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:17,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:17,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:19,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:20,790 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:22:21,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:21,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:22:22,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:23,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:25,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:27,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:27,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:28,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:29,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:31,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:32,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:33,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:33,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:35,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:36,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:22:37,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:39,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:39,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:41,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:42,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:43,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:45,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:45,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:47,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:47,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:49,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:50,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:50,790 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:22:51,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:51,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:22:53,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:55,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:56,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:22:57,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:22:57,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:22:59,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:01,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:01,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:02,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:03,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:05,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:06,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:23:07,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:07,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:09,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:11,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:12,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:12,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:13,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:15,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:17,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:17,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:18,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:19,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:20,793 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:23:21,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:21,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:23:23,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:23,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:25,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:27,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:27,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:29,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:29,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:31,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:32,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:33,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:34,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:35,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:36,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:23:37,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:39,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:40,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:41,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:42,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:43,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:45,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:45,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:47,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:47,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:49,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:50,796 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:23:51,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:51,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:51,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:23:53,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:55,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:56,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:23:57,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:23:57,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:23:59,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:01,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:02,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:02,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:03,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:05,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:06,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:24:07,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:08,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:09,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:11,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:12,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:13,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:13,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:15,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:17,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:17,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:19,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:19,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:20,801 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:24:21,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:21,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:24:23,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:24,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:25,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:27,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:27,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:29,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:30,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:31,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:32,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:33,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:35,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:35,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:36,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:24:37,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:39,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:41,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:41,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:42,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:43,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:45,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:46,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:47,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:47,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:49,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:50,801 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:24:51,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:51,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:24:52,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:53,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:55,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:57,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:24:57,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:24:57,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:24:59,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:01,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:02,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:03,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:03,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:05,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:06,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:25:07,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:08,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:09,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:11,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:12,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:13,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:14,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:15,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:17,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:17,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:19,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:19,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:20,804 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:25:21,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:21,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:25:23,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:25,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:25,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:27,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:27,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:29,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:31,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:31,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:32,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:33,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:35,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:36,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:36,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:25:37,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:39,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:41,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:42,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:42,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:43,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:45,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:47,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:47,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:47,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:49,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:50,807 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:25:51,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:52,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:25:53,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:53,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:55,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:57,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:25:57,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:25:58,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:25:59,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:01,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:02,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:03,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:04,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:05,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:07,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:26:07,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:09,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:09,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:11,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:12,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:13,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:15,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:15,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:17,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:17,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:19,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:20,810 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:26:20,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:21,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:22,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:26:23,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:25,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:26,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:27,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:27,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:29,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:31,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:31,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:32,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:33,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:35,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:37,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:26:37,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:37,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:39,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:41,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:42,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:43,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:43,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:45,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:47,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:47,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:48,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:49,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:50,812 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:26:51,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:52,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:26:53,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:54,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:26:55,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:57,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:26:57,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:59,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:26:59,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:01,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:02,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:03,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:05,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:05,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:07,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:27:07,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:09,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:10,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:11,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:12,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:13,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:15,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:16,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:17,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:17,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:19,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:20,815 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:27:21,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:21,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:22,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:27:23,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:25,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:27,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:27,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:27,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:29,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:31,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:32,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:32,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:33,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:35,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:37,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:27:37,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:38,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:39,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:41,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:42,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:43,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:43,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:45,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:47,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:47,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:49,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:49,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:50,817 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:27:51,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:52,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:27:53,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:54,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:27:55,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:57,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:27:57,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:27:59,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:00,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:01,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:02,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:03,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:05,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:06,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:07,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:28:07,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:09,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:11,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:11,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:12,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:13,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:15,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:17,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:17,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:17,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:19,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:20,820 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:28:21,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:22,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:28:22,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:23,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:25,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:27,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:27,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:28,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:29,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:31,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:32,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:33,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:33,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:35,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:37,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:28:37,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:39,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:39,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:41,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:42,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:43,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:44,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:45,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:47,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:47,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:49,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:50,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:50,823 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:28:51,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:52,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:28:53,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:55,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:55,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:28:57,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:28:57,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:28:59,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:01,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:01,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:02,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:03,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:05,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:06,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:07,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:29:07,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:09,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:11,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:12,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:12,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:13,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:15,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:17,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:17,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:18,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:19,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:20,825 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:29:21,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:22,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:29:23,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:23,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:25,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:27,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:27,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:29,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:29,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:31,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:32,746 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:33,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:34,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:35,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:37,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:29:37,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:39,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:40,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:41,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:42,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:43,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:45,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:45,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:47,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:47,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:49,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:50,829 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:29:50,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:51,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:52,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:29:53,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:55,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:56,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:29:57,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:29:57,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:29:59,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:01,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:02,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:02,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:03,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:05,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:07,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:30:07,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:07,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:09,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:11,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:12,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:13,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:13,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:15,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:17,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:17,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:18,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:19,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:20,833 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:30:21,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:22,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:30:23,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:24,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:25,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:27,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:27,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:29,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:29,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:31,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:32,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:33,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:35,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:35,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:37,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:30:37,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:39,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:41,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:41,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:42,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:43,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:45,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:46,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:47,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:47,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:49,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:50,835 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:30:51,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:52,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:30:52,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:53,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:55,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:57,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:30:57,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:30:57,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:30:59,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:01,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:02,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:03,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:03,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:05,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:07,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:31:07,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:08,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:09,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:11,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:12,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:13,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:14,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:15,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:17,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:17,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:19,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:19,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:20,838 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:31:21,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:22,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:31:23,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:25,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:25,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:27,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:27,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:29,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:30,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:31,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:32,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:33,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:35,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:36,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:37,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:31:37,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:39,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:41,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:41,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:42,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:43,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:45,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:47,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:47,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:47,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:49,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:50,844 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:31:51,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:52,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:31:52,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:53,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:55,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:57,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:31:57,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:31:58,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:31:59,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:01,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:02,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:03,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:04,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:05,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:07,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:32:07,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:09,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:09,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:11,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:12,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:13,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:15,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:15,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:17,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:17,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:19,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:20,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:20,844 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:32:21,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:22,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:32:23,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:24,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:24,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:32:25,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:25,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:27,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:27,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:29,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:30,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:31,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:32,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:33,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:35,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:35,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:37,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:32:37,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:39,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:41,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:41,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:42,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:43,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:45,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:46,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:47,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:47,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:49,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:50,847 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:32:51,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:52,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:32:52,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:53,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:55,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:57,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:32:57,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:32:58,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:32:59,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:01,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:02,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:03,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:03,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:05,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:07,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:33:07,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:09,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:09,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:11,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:12,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:13,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:14,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:15,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:17,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:17,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:19,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:20,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:20,850 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:33:21,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:22,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:33:23,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:25,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:25,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:27,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:27,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:29,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:31,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:31,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:32,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:33,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:35,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:36,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:37,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:33:37,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:39,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:41,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:42,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:42,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:43,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:45,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:47,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:47,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:47,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:49,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:50,855 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:33:51,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:52,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:33:53,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:53,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:55,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:57,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:33:57,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:33:58,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:33:59,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:01,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:02,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:03,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:04,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:05,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:07,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:34:07,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:09,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:10,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:11,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:12,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:13,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:15,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:15,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:17,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:17,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:19,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:20,855 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:34:20,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:21,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:22,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:34:23,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:25,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:26,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:27,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:27,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:29,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:31,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:32,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:32,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:33,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:35,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:37,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:34:37,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:37,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:39,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:41,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:42,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:43,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:43,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:45,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:47,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:47,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:48,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:49,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:50,858 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:34:51,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:52,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:34:53,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:54,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:34:55,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:57,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:34:58,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:34:59,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:00,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:02,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:02,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:04,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:05,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:06,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:07,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:35:08,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:10,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:10,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:12,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:12,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:14,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:16,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:16,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:17,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:18,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:20,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:20,861 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:35:21,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:22,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:35:22,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:24,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:26,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:27,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:27,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:28,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:30,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:32,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:32,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:33,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:34,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:36,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:37,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:35:38,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:40,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:42,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:42,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:44,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:44,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:46,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:47,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:48,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:49,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:50,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:50,946 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:35:52,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:35:52,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:54,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:55,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:35:56,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:35:57,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:35:58,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:00,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:00,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:02,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:02,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:04,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:06,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:06,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:07,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:36:08,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:10,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:11,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:12,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:12,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:14,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:16,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:17,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:17,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:18,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:20,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:20,866 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:36:22,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:36:22,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:22,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:24,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:26,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:27,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:28,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:28,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:30,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:32,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:32,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:33,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:34,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:36,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:37,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:36:38,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:39,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:40,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:42,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:42,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:44,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:44,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:46,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:47,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:48,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:50,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:50,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:50,869 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:36:52,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:36:52,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:54,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:56,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:36:56,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:36:57,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:36:58,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:00,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:01,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:02,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:02,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:04,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:06,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:07,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:37:07,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:08,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:10,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:12,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:12,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:12,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:14,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:16,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:17,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:18,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:18,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:20,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:20,871 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:37:22,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:37:22,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:23,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:24,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:26,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:27,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:28,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:29,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:30,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:32,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:32,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:34,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:34,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:36,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:37,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:37:38,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:40,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:40,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:42,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:42,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:44,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:45,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:46,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:47,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:48,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:50,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:50,876 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:37:51,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:52,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:37:52,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:54,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:56,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:37:56,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:37:57,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:37:58,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:00,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:02,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:02,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:02,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:04,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:06,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:07,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:38:07,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:08,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:10,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:12,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:12,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:13,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:14,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:16,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:17,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:18,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:19,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:20,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:20,878 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:38:22,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:38:22,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:24,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:24,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:26,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:27,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:28,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:30,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:30,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:32,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:32,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:34,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:35,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:36,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:37,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:38:38,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:40,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:41,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:42,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:42,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:44,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:46,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:46,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:47,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:48,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:50,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:50,880 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:38:52,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:38:52,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:52,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:54,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:56,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:38:57,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:38:57,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:38:58,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:00,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:02,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:02,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:03,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:04,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:06,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:07,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:39:08,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:08,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:10,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:12,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:12,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:14,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:14,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:16,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:17,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:18,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:19,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:20,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:20,883 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:39:22,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:39:22,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:24,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:25,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:26,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:27,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:28,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:30,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:31,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:32,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:32,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:34,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:36,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:36,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:37,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:39:38,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:40,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:42,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:42,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:42,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:44,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:46,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:47,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:47,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:48,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:50,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:50,887 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:39:52,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:39:52,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:53,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:39:54,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:56,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:57,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:39:58,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:39:58,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:00,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:02,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:02,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:04,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:04,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:06,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:07,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:40:08,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:09,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:10,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:12,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:12,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:14,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:15,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:16,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:17,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:18,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:20,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:20,889 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:40:20,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:22,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:40:22,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:24,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:26,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:26,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:27,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:28,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:30,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:31,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:32,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:32,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:34,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:36,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:37,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:40:37,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:38,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:40,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:42,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:42,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:43,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:44,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:46,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:47,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:48,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:48,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:50,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:50,893 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:40:52,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:40:52,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:54,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:40:54,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:56,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:57,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:40:58,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:40:59,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:00,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:02,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:02,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:04,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:05,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:06,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:07,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:41:08,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:10,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:10,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:12,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:12,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:14,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:16,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:16,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:17,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:18,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:20,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:20,896 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:41:21,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:22,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:41:22,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:24,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:26,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:27,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:27,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:28,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:30,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:32,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:32,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:32,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:34,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:36,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:37,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:41:38,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:38,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:40,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:42,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:42,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:43,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:44,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:46,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:47,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:48,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:49,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:50,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:50,898 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:41:52,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:41:52,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:54,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:54,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:41:56,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:41:57,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:41:58,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:00,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:00,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:02,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:02,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:04,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:06,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:06,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:07,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:42:08,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:10,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:11,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:12,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:12,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:14,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:16,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:17,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:17,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:18,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:20,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:20,901 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:42:22,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:42:22,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:22,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:24,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:26,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:27,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:28,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:28,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:30,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:32,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:32,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:33,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:34,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:36,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:37,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:42:38,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:39,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:40,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:42,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:42,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:44,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:44,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:46,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:47,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:48,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:50,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:50,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:50,904 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:42:52,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:42:52,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:54,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:55,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:42:56,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:42:57,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:42:58,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:00,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:01,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:02,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:02,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:04,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:06,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:06,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:07,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:43:08,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:10,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:12,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:12,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:12,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:14,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:16,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:17,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:18,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:18,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:20,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:20,909 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:43:22,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:43:22,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:23,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:24,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:26,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:27,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:28,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:29,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:30,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:32,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:32,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:34,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:34,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:36,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:37,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:43:38,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:40,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:40,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:42,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:42,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:44,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:45,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:46,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:47,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:48,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:50,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:50,910 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:43:50,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:52,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:43:52,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:54,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:56,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:43:56,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:43:57,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:43:58,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:00,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:02,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:02,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:02,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:04,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:06,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:07,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:44:07,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:08,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:10,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:12,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:12,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:13,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:14,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:16,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:17,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:18,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:18,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:20,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:20,913 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:44:22,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:44:22,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:24,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:24,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:26,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:27,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:28,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:29,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:30,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:32,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:32,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:34,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:35,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:36,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:37,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:44:38,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:40,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:40,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:42,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:42,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:44,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:46,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:46,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:47,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:48,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:50,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:50,915 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:44:52,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:44:52,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:52,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:54,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:56,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:44:57,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:44:57,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:44:58,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:00,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:02,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:02,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:03,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:04,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:06,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:07,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:45:08,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:08,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:10,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:12,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:12,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:14,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:14,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:16,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:17,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:18,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:19,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:20,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:20,922 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:45:22,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:45:22,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:24,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:25,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:26,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:27,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:28,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:30,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:30,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:32,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:32,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:34,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:36,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:36,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:37,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:45:38,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:40,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:41,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:42,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:42,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:44,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:46,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:47,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:47,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:48,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:50,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:50,921 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:45:52,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:45:52,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:52,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:54,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:56,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:45:57,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:45:58,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:45:58,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:00,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:02,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:02,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:04,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:04,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:06,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:07,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:46:08,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:09,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:10,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:12,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:12,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:14,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:15,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:16,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:17,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:18,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:20,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:20,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:20,924 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:46:22,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:46:22,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:24,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:26,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:26,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:27,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:28,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:30,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:31,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:32,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:32,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:34,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:36,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:37,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:46:37,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:38,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:40,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:42,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:42,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:42,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:44,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:46,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:47,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:48,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:48,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:50,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:50,926 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:46:52,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:46:52,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:53,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:46:54,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:56,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:57,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:46:58,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:46:59,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:00,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:02,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:02,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:04,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:04,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:06,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:07,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:47:08,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:10,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:10,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:12,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:12,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:14,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:15,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:16,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:17,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:18,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:20,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:20,931 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:47:21,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:47:22,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:24,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:26,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:27,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:27,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:28,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:30,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:32,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:32,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:32,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:34,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:36,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:37,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:47:38,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:38,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:40,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:42,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:42,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:43,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:44,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:46,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:47,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:48,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:49,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:50,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:50,932 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:47:52,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:47:52,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:54,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:47:54,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:56,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:47:57,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:47:58,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:00,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:00,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:02,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:02,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:04,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:05,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:06,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:07,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:48:08,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:10,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:11,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:12,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:12,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:14,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:16,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:16,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:17,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:18,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:20,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:20,934 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:48:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:48:22,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:22,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:24,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:26,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:27,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:27,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:28,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:30,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:32,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:32,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:33,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:34,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:36,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:37,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:48:38,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:38,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:40,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:42,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:42,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:44,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:44,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:46,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:47,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:48,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:50,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:50,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:50,937 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:48:52,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:48:52,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:54,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:55,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:48:56,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:48:57,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:48:58,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:00,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:01,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:02,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:02,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:04,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:06,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:06,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:07,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:49:08,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:10,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:12,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:12,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:12,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:14,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:16,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:17,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:17,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:18,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:20,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:20,940 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:49:22,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:49:22,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:23,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:24,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:26,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:27,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:28,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:28,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:30,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:32,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:32,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:34,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:34,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:36,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:37,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:49:38,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:39,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:40,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:42,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:42,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:44,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:45,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:46,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:47,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:48,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:50,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:50,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:50,945 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:49:52,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:49:52,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:54,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:56,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:49:56,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:49:57,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:49:58,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:00,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:01,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:02,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:02,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:04,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:06,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:07,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:50:07,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:08,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:10,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:12,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:12,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:13,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:14,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:16,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:17,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:18,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:18,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:20,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:20,947 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:50:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:50:22,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:24,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:24,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:26,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:27,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:28,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:29,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:30,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:32,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:32,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:34,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:35,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:36,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:37,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:50:38,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:40,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:40,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:42,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:42,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:44,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:46,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:46,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:47,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:48,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:50,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:50,965 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:50:51,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:52,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:50:52,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:54,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:56,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:50:57,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:50:57,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:50:58,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:00,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:02,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:02,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:02,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:04,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:06,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:07,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:51:08,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:08,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:10,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:12,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:12,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:13,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:14,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:16,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:17,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:18,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:19,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:20,953 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:51:20,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:22,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:51:22,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:24,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:25,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:26,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:27,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:28,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:30,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:30,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:32,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:32,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:34,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:36,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:36,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:37,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:51:38,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:40,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:41,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:42,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:42,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:44,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:46,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:47,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:47,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:48,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:50,958 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:51:50,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:52,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:51:52,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:52,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:54,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:56,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:51:57,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:51:58,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:51:58,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:01,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:02,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:03,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:03,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:05,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:07,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:07,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:52:09,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:09,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:11,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:12,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:13,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:14,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:15,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:17,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:17,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:19,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:20,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:20,960 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:52:21,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:22,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:52:23,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:25,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:26,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:27,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:27,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:29,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:31,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:31,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:32,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:33,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:35,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:37,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:52:37,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:37,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:39,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:41,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:42,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:42,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:43,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:45,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:47,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:47,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:48,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:49,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:50,965 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:52:51,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:52,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:52:53,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:53,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:52:55,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:57,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:57,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:52:59,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:52:59,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:01,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:02,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:03,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:04,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:05,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:07,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:53:07,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:09,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:10,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:11,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:12,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:13,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:15,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:15,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:17,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:17,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:19,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:20,966 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:53:20,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:21,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:22,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:53:23,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:25,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:26,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:27,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:27,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:29,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:31,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:32,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:32,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:33,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:35,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:37,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:53:37,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:37,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:39,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:41,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:42,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:43,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:43,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:45,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:47,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:47,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:49,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:49,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:50,969 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:53:51,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:52,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:53:53,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:54,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:53:55,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:57,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:53:57,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:53:59,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:00,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:01,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:02,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:03,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:05,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:05,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:07,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:54:07,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:09,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:11,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:11,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:12,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:13,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:15,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:16,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:17,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:17,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:19,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:20,971 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:54:21,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:22,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:54:22,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:23,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:25,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:27,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:27,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:29,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:31,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:32,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:33,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:33,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:35,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:37,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:54:37,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:38,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:39,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:41,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:42,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:43,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:44,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:45,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:47,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:47,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:49,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:49,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:50,974 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:54:51,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:52,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:54:53,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:55,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:55,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:54:57,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:54:57,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:54:59,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:00,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:01,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:02,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:03,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:05,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:06,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:07,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:55:07,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:09,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:11,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:12,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:12,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:13,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:15,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:17,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:17,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:17,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:19,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:21,003 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:55:21,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:22,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:55:23,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:23,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:25,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:27,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:28,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:28,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:28,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:29,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:29,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 03:55:31,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:32,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:33,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:34,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:35,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:37,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:55:37,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:39,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:39,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:41,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:42,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:43,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:45,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:45,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:47,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:47,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:49,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:50,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:51,006 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:55:51,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:52,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:55:53,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:55,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:56,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:55:57,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:55:57,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:55:59,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:01,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:01,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:02,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:03,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:05,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:07,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:56:07,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:07,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:09,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:11,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:12,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:12,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:13,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:15,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:17,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:17,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:18,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:19,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:21,008 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:56:21,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:22,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:56:23,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:24,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:25,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:27,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:29,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:29,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:31,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:32,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:33,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:35,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:35,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:37,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:56:37,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:39,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:40,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:41,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:42,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:43,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:45,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:46,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:47,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:47,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:49,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:51,012 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:56:51,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:52,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:56:52,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:53,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:55,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:57,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:56:57,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:56:57,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:56:59,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:01,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:02,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:03,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:03,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:05,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:07,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:57:07,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:08,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:09,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:11,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:12,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:13,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:13,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:15,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:17,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:18,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:19,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:19,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:21,015 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:57:21,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:22,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:57:23,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:24,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:25,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:27,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:29,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:30,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:31,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:33,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:33,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:35,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:35,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:37,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:57:37,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:39,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:41,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:41,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:42,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:43,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:45,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:47,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:47,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:48,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:49,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:51,018 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:57:51,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:52,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:57:52,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:53,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:55,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:57,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:57:57,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:57:58,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:57:59,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:01,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:03,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:03,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:03,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:05,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:07,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:58:07,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:09,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:09,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:11,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:12,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:13,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:14,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:15,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:17,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:18,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:19,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:20,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:21,023 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:58:21,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:22,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:58:23,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:25,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:25,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:27,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:27,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:29,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:31,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:31,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:33,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:33,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:35,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:36,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:37,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:58:37,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:39,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:41,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:42,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:42,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:43,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:45,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:47,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:47,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:47,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:49,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:51,024 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:58:51,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:52,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:58:53,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:53,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:55,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:57,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:58:57,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:58:58,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:58:59,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:01,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:02,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:03,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:04,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:05,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:07,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:59:07,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:09,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:10,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:11,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:12,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:13,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:15,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:15,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:17,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:17,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:19,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:21,027 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:59:21,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:21,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:22,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:59:23,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:25,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:26,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:27,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:27,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:29,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:31,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:32,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:32,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:33,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:35,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:37,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:59:37,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:37,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:39,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:41,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:42,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:42,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:43,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:45,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:47,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:47,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:47,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:49,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:51,030 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 03:59:51,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:52,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 03:59:53,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:53,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:55,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:57,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 03:59:57,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 03:59:58,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 03:59:59,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:01,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:02,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:03,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:03,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:05,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:07,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:00:07,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:09,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:09,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:11,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:12,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:13,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:14,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:15,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:17,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:17,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:19,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:19,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:21,033 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:00:21,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:22,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:00:23,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:23,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:00:23,834 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 04:00:23,842 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 04:00:23,852 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 04:00:24,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 04:00:24,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:25,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:27,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:28,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:29,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:33,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:34,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:37,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:00:37,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:40,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:43,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:45,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:47,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:00:47,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:00:47,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 04:00:47,373 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 04:00:47,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:00:47,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_9_570b2d39ebd6823f0cfe.table.json 2025-09-18 04:00:47,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 04:00:48,036 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/75h69h6e-media/table/rollouts_9_570b2d39ebd6823f0cfe.table.json 2025-09-18 04:00:48,285 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmp40ataztd 2025-09-18 04:00:48,899 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MjIyNTQwOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MjExMDU5MA==', 'versionIndex': 7}}} 2025-09-18 04:00:48,899 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 04:00:48,899 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_9_570b2d39ebd6823f0cfe.table.json with policy now 2025-09-18 04:00:49,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:49,244 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/1hsn57j4-media/table/rollouts_9_570b2d39ebd6823f0cfe.table.json 2025-09-18 04:00:49,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:51,037 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:00:51,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:51,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:52,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:00:53,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:55,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:56,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:00:57,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:00:57,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:00:59,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:01,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:01,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:03,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:03,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:05,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:07,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:01:07,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:07,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:09,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:11,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:12,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:12,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:13,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:15,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:17,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:18,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:18,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:19,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:21,039 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:01:21,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:22,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:01:23,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:23,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:25,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:27,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:27,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:29,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:29,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:31,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:33,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:33,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:34,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:35,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:37,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:01:37,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:39,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:40,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:41,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:42,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:43,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:45,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:46,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:47,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:48,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:49,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:51,042 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:01:51,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:51,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:52,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:01:53,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:55,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:56,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:01:57,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:01:57,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:01:59,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:01,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:01,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:03,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:03,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:05,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:06,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:07,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:02:07,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:09,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:11,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:12,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:12,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:13,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:15,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:17,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:17,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:18,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:19,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:21,046 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:02:21,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:22,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:02:23,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:23,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:25,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:27,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:27,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:28,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:29,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:31,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:33,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:33,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:34,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:35,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:37,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:02:37,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:39,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:39,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:41,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:42,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:43,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:45,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:45,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:47,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:48,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:49,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:51,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:51,052 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:02:51,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:52,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:02:53,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:55,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:56,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:02:57,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:02:57,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:02:59,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:01,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:02,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:03,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:03,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:05,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:07,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:03:07,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:07,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:09,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:11,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:12,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:13,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:13,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:15,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:17,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:18,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:18,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:19,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:21,052 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:03:21,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:22,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:03:23,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:24,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:25,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:27,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:27,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:29,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:29,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:31,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:33,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:33,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:35,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:35,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:37,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:03:37,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:39,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:40,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:41,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:42,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:43,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:45,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:46,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:47,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:47,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:49,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:51,056 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:03:51,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:52,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:52,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:03:53,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:55,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:57,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:03:57,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:03:57,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:03:59,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:01,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:02,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:02,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:03,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:05,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:07,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:04:07,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:08,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:09,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:11,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:12,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:13,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:14,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:15,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:17,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:17,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:19,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:19,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:21,058 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:04:21,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:22,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:04:23,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:25,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:25,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:27,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:27,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:29,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:30,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:31,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:32,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:33,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:35,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:36,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:37,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:04:37,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:39,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:41,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:41,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:42,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:43,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:45,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:47,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:47,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:47,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:49,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:51,061 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:04:51,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:52,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:04:52,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:53,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:55,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:57,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:04:57,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:04:58,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:04:59,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:01,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:02,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:03,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:03,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:05,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:07,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:05:07,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:09,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:09,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:11,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:12,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:13,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:14,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:15,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:17,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:17,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:19,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:20,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:21,064 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:05:21,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:22,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:05:23,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:25,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:26,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:27,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:27,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:29,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:31,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:31,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:32,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:33,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:35,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:37,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:05:37,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:37,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:39,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:41,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:42,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:42,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:43,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:45,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:47,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:47,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:48,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:49,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:51,067 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:05:51,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:52,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:05:53,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:53,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:55,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:57,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:05:57,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:05:59,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:05:59,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:01,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:02,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:03,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:04,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:05,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:07,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:06:07,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:09,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:10,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:11,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:12,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:13,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:15,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:15,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:17,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:17,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:19,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:21,070 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:06:21,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:21,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:22,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:06:23,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:25,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:26,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:27,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:27,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:29,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:31,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:32,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:32,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:33,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:35,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:37,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:06:37,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:37,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:39,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:41,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:42,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:43,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:43,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:45,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:47,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:47,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:49,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:49,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:51,073 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:06:51,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:52,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:06:53,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:54,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:06:55,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:57,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:06:57,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:06:59,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:00,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:01,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:02,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:03,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:05,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:05,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:07,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:07:07,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:09,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:11,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:11,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:12,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:13,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:15,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:16,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:17,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:17,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:19,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:21,075 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:07:21,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:22,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:07:22,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:23,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:25,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:27,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:27,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:27,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:29,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:31,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:32,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:33,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:33,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:35,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:37,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:07:37,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:38,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:39,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:41,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:42,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:43,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:44,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:45,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:47,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:47,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:49,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:49,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:51,078 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:07:51,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:52,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:07:53,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:55,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:07:55,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:57,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:07:57,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:07:59,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:00,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:01,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:02,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:03,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:05,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:06,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:07,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:08:07,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:09,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:11,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:12,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:12,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:13,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:15,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:17,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:17,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:17,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:19,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:21,082 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:08:21,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:22,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:08:23,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:23,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:25,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:27,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:27,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:28,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:29,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:31,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:32,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:33,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:34,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:35,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:37,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:08:37,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:39,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:39,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:41,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:42,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:43,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:45,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:45,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:47,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:47,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:49,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:50,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:51,084 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:08:51,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:52,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:08:53,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:55,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:08:56,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:08:57,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:08:58,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:00,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:01,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:02,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:02,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:04,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:06,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:07,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:09:07,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:08,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:10,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:12,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:12,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:12,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:14,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:16,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:17,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:18,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:18,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:20,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:21,087 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:09:22,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:22,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:09:23,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:24,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:26,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:27,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:28,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:29,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:30,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:32,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:32,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:34,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:34,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:36,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:37,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:09:38,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:40,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:40,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:42,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:42,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:44,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:46,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:46,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:47,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:48,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:50,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:51,090 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:09:51,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:52,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:52,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:09:54,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:56,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:09:57,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:09:57,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:09:58,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:00,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:02,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:02,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:02,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:04,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:06,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:07,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:10:08,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:08,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:10,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:12,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:12,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:13,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:14,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:16,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:17,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:18,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:19,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:20,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:21,094 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:10:22,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:10:22,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:24,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:24,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:26,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:27,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:28,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:30,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:30,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:32,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:32,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:34,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:35,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:36,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:37,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:10:38,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:40,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:41,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:42,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:42,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:44,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:46,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:46,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:47,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:48,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:50,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:51,095 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:10:52,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:10:52,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:52,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:54,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:56,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:10:57,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:10:57,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:10:58,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:00,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:02,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:02,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:03,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:04,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:06,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:07,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:11:08,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:09,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:10,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:12,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:12,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:14,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:14,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:16,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:17,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:18,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:20,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:20,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:21,099 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:11:22,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:11:22,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:24,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:25,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:26,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:27,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:28,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:30,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:31,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:32,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:32,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:34,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:36,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:36,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:37,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:11:38,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:40,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:42,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:42,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:42,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:44,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:46,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:47,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:47,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:48,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:50,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:51,101 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:11:52,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:11:52,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:53,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:11:54,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:56,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:57,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:11:58,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:11:58,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:00,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:02,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:02,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:04,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:04,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:06,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:07,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:12:08,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:09,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:10,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:12,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:12,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:14,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:15,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:16,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:17,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:18,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:20,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:20,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:21,107 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:12:22,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:12:22,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:24,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:26,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:26,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:27,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:28,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:30,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:32,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:32,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:32,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:34,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:36,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:37,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:12:37,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:38,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:40,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:42,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:42,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:43,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:44,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:46,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:47,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:48,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:48,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:50,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:51,106 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:12:52,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:12:52,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:54,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:12:54,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:56,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:57,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:12:58,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:12:59,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:00,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:02,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:02,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:04,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:05,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:06,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:07,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:13:08,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:10,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:10,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:12,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:12,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:14,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:16,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:16,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:17,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:18,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:20,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:21,109 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:13:22,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:13:22,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:22,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:24,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:26,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:27,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:27,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:28,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:30,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:32,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:32,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:32,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:34,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:36,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:37,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:13:38,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:38,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:40,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:42,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:42,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:43,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:44,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:46,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:47,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:48,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:49,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:50,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:51,112 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:13:52,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:13:52,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:54,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:55,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:13:56,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:13:57,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:13:58,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:00,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:00,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:02,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:02,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:04,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:06,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:06,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:07,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:14:08,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:10,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:11,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:12,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:12,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:14,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:16,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:17,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:17,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:18,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:20,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:21,117 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:14:22,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:14:22,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:22,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:24,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:26,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:27,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:28,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:28,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:30,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:32,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:32,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:33,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:34,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:36,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:37,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:14:38,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:39,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:40,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:42,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:42,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:44,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:44,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:46,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:47,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:48,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:50,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:50,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:51,118 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:14:52,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:14:52,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:54,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:55,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:14:56,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:14:57,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:14:58,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:00,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:01,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:02,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:02,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:04,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:06,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:06,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:07,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:15:08,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:10,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:12,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:12,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:12,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:14,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:16,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:17,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:18,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:18,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:20,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:21,121 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:15:22,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:15:22,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:23,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:24,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:26,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:27,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:28,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:29,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:30,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:32,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:32,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:34,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:34,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:36,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:37,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:15:38,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:40,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:40,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:42,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:42,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:44,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:45,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:46,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:47,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:48,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:50,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:51,124 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:15:51,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:52,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:15:52,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:54,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:56,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:15:56,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:15:57,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:15:58,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:00,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:02,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:02,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:02,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:04,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:06,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:07,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:16:07,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:08,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:10,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:12,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:12,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:13,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:14,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:16,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:17,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:18,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:18,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:20,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:21,126 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:16:22,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:16:22,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:24,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:24,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:26,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:27,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:28,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:30,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:30,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:32,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:32,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:34,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:35,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:36,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:37,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:16:38,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:40,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:41,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:42,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:42,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:44,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:46,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:46,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:47,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:48,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:50,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:51,129 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:16:52,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:16:52,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:52,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:54,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:56,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:16:57,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:16:57,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:16:58,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:00,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:02,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:02,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:03,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:04,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:06,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:07,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:17:08,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:08,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:10,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:12,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:12,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:14,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:14,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:16,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:17,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:18,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:19,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:20,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:21,132 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:17:22,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:17:22,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:24,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:25,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:26,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:27,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:28,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:30,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:30,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:32,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:32,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:34,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:36,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:36,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:37,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:17:38,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:40,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:41,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:42,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:42,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:44,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:46,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:47,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:47,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:48,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:50,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:51,135 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:17:52,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:17:52,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:53,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:17:54,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:56,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:57,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:17:58,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:17:58,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:00,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:02,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:02,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:04,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:04,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:06,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:07,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:18:08,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:09,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:10,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:12,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:12,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:14,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:15,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:16,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:17,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:18,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:20,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:20,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:21,137 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:18:22,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:18:22,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:24,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:26,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:26,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:27,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:28,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:30,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:31,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:32,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:32,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:34,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:36,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:37,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:18:37,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:38,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:40,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:42,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:42,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:42,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:44,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:46,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:47,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:48,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:48,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:50,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:51,140 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:18:52,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:18:52,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:53,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:18:54,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:56,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:57,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:18:58,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:18:59,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:00,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:02,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:02,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:04,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:04,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:06,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:07,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:19:08,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:10,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:10,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:12,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:12,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:14,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:16,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:16,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:17,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:18,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:20,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:21,144 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:19:21,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:22,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:19:22,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:24,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:26,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:27,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:27,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:28,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:30,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:32,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:32,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:32,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:34,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:36,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:37,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:19:38,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:38,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:38,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,746 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:38,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:39,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:39,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:39,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:19:40,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:42,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:42,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:43,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:44,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:46,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:47,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:48,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:49,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:50,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:51,149 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:19:52,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:19:52,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:54,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:54,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:19:56,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:19:57,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:19:58,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:00,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:00,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:02,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:02,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:04,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:05,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:06,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:07,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:20:08,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:10,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:11,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:12,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:12,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:14,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:16,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:17,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:17,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:18,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:20,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:21,149 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:20:22,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:20:22,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:22,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:24,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:26,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:27,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:28,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:28,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:30,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:32,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:32,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:33,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:34,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:36,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:37,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:20:38,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:39,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:40,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:42,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:42,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:44,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:44,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:46,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:47,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:48,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:50,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:50,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:51,152 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:20:52,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:20:52,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:54,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:55,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:20:56,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:20:57,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:20:58,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:00,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:01,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:02,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:02,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:04,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:06,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:06,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:07,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:21:08,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:10,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:12,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:12,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:12,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:14,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:16,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:17,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:17,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:18,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:20,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:21,155 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:21:22,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:21:22,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:23,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:25,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:27,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:27,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:28,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:29,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:31,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:32,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:33,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:34,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:35,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:37,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:21:37,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:39,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:39,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:41,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:42,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:43,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:45,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:45,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:47,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:47,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:49,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:51,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:51,159 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:21:51,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:52,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:21:53,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:55,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:56,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:21:57,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:21:57,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:21:59,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:01,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:02,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:02,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:03,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:05,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:07,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:22:07,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:07,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:09,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:11,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:12,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:13,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:13,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:15,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:17,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:17,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:18,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:19,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:21,160 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:22:21,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:22,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:22:23,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:24,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:25,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:27,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:27,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:29,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:29,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:31,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:32,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:33,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:35,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:35,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:37,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:22:37,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:39,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:40,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:41,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:42,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:43,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:45,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:46,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:47,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:47,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:49,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:51,163 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:22:51,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:52,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:22:52,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:53,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:55,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:57,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:22:57,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:22:57,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:22:59,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:01,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:02,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:02,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:03,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:05,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:07,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:23:07,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:08,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:09,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:11,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:12,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:13,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:14,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:15,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:17,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:17,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:19,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:19,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:21,165 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:23:21,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:22,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:23:23,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:25,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:25,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:27,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:27,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:29,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:30,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:31,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:32,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:33,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:35,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:36,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:37,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:23:37,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:39,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:41,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:41,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:42,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:43,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:45,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:47,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:47,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:47,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:49,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:51,171 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:23:51,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:52,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:23:52,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:53,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:55,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:57,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:23:57,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:23:58,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:23:59,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:01,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:02,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:03,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:03,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:05,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:07,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:24:07,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:09,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:09,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:11,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:12,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:13,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:14,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:15,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:17,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:17,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:19,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:20,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:21,171 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:24:21,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:22,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:24:23,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:25,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:25,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:27,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:27,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:29,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:31,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:31,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:32,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:33,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:35,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:37,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:37,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:24:37,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:39,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:41,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:42,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:42,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:43,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:45,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:47,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:47,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:48,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:49,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:51,175 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:24:51,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:52,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:24:53,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:53,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:55,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:57,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:24:57,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:24:59,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:24:59,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:01,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:02,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:03,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:04,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:05,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:07,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:25:07,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:09,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:10,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:11,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:12,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:13,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:15,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:15,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:17,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:17,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:19,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:21,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:21,179 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:25:21,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:22,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:25:23,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:25,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:26,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:27,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:27,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:29,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:31,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:32,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:32,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:33,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:35,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:37,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:25:37,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:37,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:39,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:41,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:42,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:43,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:43,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:45,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:47,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:47,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:48,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:49,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:51,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:51,182 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:25:52,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:25:53,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:54,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:25:55,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:57,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:25:57,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:25:59,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:00,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:01,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:02,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:03,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:05,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:05,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:07,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:26:07,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:09,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:11,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:11,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:12,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:13,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:15,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:16,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:17,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:17,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:19,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:21,187 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:26:21,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:22,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:26:22,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:23,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:25,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:27,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:27,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:27,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:29,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:31,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:32,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:33,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:33,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:35,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:37,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:26:37,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:38,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:39,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:41,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:42,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:43,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:44,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:45,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:47,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:47,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:49,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:49,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:51,189 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:26:51,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:52,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:26:53,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:55,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:55,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:26:57,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:26:57,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:26:59,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:00,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:01,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:02,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:03,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:05,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:06,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:07,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:27:07,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:09,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:11,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:12,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:12,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:13,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:15,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:17,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:17,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:17,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:19,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:21,193 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:27:21,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:22,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:27:23,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:23,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:25,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:27,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:27,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:28,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:29,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:31,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:32,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:33,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:34,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:35,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:37,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:27:37,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:39,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:39,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:41,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:42,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:43,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:45,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:45,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:47,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:47,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:49,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:50,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:51,195 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:27:51,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:52,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:27:53,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:55,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:56,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:27:57,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:27:57,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:27:59,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:01,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:01,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:02,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:03,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:05,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:07,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:28:07,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:07,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:09,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:11,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:12,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:12,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:13,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:15,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:17,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:17,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:18,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:19,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:21,197 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:28:21,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:22,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:28:23,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:24,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:25,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:27,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:27,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:27,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:29,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:29,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:31,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:32,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:33,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:35,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:35,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:37,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:28:37,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:39,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:40,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:41,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:42,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:43,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:45,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:46,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:47,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:47,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:49,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:51,201 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:28:51,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:51,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:52,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:28:53,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:55,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:56,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:28:57,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:28:57,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:28:59,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:01,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:01,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:02,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:03,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:05,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:06,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:07,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:29:07,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:09,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:11,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:12,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:12,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:13,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:15,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:17,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:17,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:17,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:19,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:21,206 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:29:21,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:22,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:29:23,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:23,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:25,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:27,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:27,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:29,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:29,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:31,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:32,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:33,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:34,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:35,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:37,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:29:37,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:39,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:40,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:41,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:42,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:43,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:45,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:45,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:47,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:47,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:49,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:51,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:51,206 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:29:51,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:52,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:29:53,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:55,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:56,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:29:57,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:29:57,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:29:59,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:01,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:02,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:02,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:03,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:05,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:07,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:30:07,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:07,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:09,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:11,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:12,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:13,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:13,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:15,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:17,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:17,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:18,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:19,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:21,209 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:30:21,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:22,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:30:23,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:24,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:25,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:27,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:27,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:29,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:29,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:31,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:32,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:33,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:35,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:35,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:37,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:30:37,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:39,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:40,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:41,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:42,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:43,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:45,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:46,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:47,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:47,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:49,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:51,211 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:30:51,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:52,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:30:52,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:53,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:55,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:57,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:30:57,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:30:57,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:30:59,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:01,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:02,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:03,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:03,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:05,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:07,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:31:07,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:08,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:09,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:11,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:12,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:13,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:14,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:15,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:17,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:17,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:19,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:19,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:21,216 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:31:21,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:22,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:31:23,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:25,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:25,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:27,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:27,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:29,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:30,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:31,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:32,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:33,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:35,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:36,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:37,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:31:37,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:39,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:41,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:41,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:42,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:43,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:45,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:47,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:47,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:47,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:49,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:51,217 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:31:51,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:52,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:31:52,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:53,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:55,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:57,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:31:57,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:31:58,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:31:59,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:01,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:02,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:03,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:03,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:05,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:07,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:32:07,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:09,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:09,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:11,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:12,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:13,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:15,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:15,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:17,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:17,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:19,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:20,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:21,220 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:32:21,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:22,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:32:23,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:25,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:26,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:27,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:27,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:29,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:31,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:31,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:32,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:33,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:35,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:37,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:32:37,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:37,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:39,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:41,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:42,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:42,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:43,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:45,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:47,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:47,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:48,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:49,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:51,222 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:32:51,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:52,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:32:53,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:53,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:55,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:57,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:32:57,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:32:59,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:32:59,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:01,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:02,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:03,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:04,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:05,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:07,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:33:07,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:09,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:10,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:11,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:12,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:13,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:15,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:15,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:17,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:17,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:19,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:21,225 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:33:21,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:21,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:22,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:33:23,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:25,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:26,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:27,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:27,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:29,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:31,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:32,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:32,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:33,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:35,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:37,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:33:37,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:37,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:39,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:41,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:42,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:43,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:43,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:45,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:47,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:47,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:48,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:49,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:51,228 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:33:51,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:52,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:33:53,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:54,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:33:55,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:57,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:33:57,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:33:59,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:00,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:01,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:02,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:03,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:05,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:05,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:07,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:34:07,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:09,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:11,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:11,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:12,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:14,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:16,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:16,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:17,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:18,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:20,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:21,231 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:34:22,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:34:22,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:22,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:24,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:26,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:27,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:27,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:28,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:30,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:32,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:32,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:33,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:34,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:36,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:37,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:34:38,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:38,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:40,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:42,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:42,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:44,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:44,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:46,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:47,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:48,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:49,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:50,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:51,234 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:34:52,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:34:52,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:54,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:55,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:34:56,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:34:57,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:34:58,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:00,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:00,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:02,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:02,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:04,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:06,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:06,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:07,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:35:08,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:10,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:12,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:12,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:12,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:14,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:16,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:17,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:17,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:18,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:20,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:21,236 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:35:22,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:35:22,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:23,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:24,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:26,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:27,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:28,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:28,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:30,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:32,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:32,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:34,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:34,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:36,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:37,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:35:38,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:39,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:40,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:42,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:42,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:44,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:45,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:46,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:47,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:48,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:50,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:50,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:51,239 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:35:52,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:35:52,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:54,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:56,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:35:56,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:35:57,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:35:58,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:00,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:01,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:02,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:02,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:04,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:06,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:07,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:36:07,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:08,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:10,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:12,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:12,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:12,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:14,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:16,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:17,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:18,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:18,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:20,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:21,242 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:36:22,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:36:22,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:23,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:24,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:26,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:27,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:28,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:29,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:30,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:32,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:32,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:34,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:35,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:36,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:37,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:36:38,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:40,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:40,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:42,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:42,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:44,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:46,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:46,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:47,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:48,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:50,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:51,247 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:36:51,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:52,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:36:52,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:54,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:56,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:36:57,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:36:57,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:36:58,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:00,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:02,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:02,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:02,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:04,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:06,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:07,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:37:08,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:08,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:10,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:12,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:12,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:13,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:14,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:16,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:17,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:18,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:19,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:20,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:21,247 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:37:22,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:37:22,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:24,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:24,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:26,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:27,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:28,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:30,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:30,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:32,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:32,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:34,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:35,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:36,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:37,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:37:38,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:40,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:41,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:42,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:42,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:44,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:46,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:47,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:47,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:48,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:50,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:51,250 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:37:52,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:37:52,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:52,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:54,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:56,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:37:57,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:37:58,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:37:58,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:00,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:02,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:02,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:03,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:04,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:06,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:07,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:38:08,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:09,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:10,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:12,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:12,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:14,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:14,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:16,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:17,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:18,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:20,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:20,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:21,253 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:38:22,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:38:22,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:24,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:25,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:26,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:27,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:28,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:30,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:31,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:32,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:32,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:34,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:36,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:36,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:37,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:38:38,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:40,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:42,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:42,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:42,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:44,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:46,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:47,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:47,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:48,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:50,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:51,256 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:38:52,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:38:52,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:53,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:54,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:56,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:38:57,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:38:58,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:38:58,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:00,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:02,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:02,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:04,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:04,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:06,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:07,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:39:08,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:09,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:10,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:12,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:12,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:14,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:15,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:16,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:17,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:18,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:20,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:21,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:21,258 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:39:22,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:39:22,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:24,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:26,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:26,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:27,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:28,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:30,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:32,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:32,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:32,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:34,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:36,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:37,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:39:37,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:38,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:40,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:42,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:42,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:43,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:44,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:46,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:47,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:48,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:48,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:50,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:51,261 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:39:52,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:39:52,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:53,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:54,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:56,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:39:57,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:39:58,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:39:58,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:00,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:02,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:02,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:04,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:04,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:06,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:07,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:40:08,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:09,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:10,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:12,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:12,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:14,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:14,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:16,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:17,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:18,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:20,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:20,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:21,264 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:40:22,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:40:22,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:24,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:25,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:27,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:28,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:40:28,221 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 04:40:28,228 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 04:40:28,238 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 04:40:28,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 04:40:30,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:31,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:33,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:36,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:37,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:40:42,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:43,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:43,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:47,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:48,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:51,299 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:40:51,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:40:51,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:40:51,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 04:40:51,838 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 04:40:51,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:40:52,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_10_1700d5d5174101ae4c4f.table.json 2025-09-18 04:40:52,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 04:40:52,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:40:52,533 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/jrznm5a4-media/table/rollouts_10_1700d5d5174101ae4c4f.table.json 2025-09-18 04:40:52,767 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpydpbkqo7 2025-09-18 04:40:53,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:53,504 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MjM1ODMwOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MjIyNTQwOQ==', 'versionIndex': 8}}} 2025-09-18 04:40:53,504 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 04:40:53,504 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_10_1700d5d5174101ae4c4f.table.json with policy now 2025-09-18 04:40:53,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:40:53,822 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/9cq08rrw-media/table/rollouts_10_1700d5d5174101ae4c4f.table.json 2025-09-18 04:40:55,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:57,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:57,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:40:59,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:40:59,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:01,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:02,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:03,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:04,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:05,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:07,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:07,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:41:09,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:10,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:11,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:12,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:13,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:15,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:15,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:17,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:17,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:19,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:21,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:21,302 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:41:21,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:22,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:41:23,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:25,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:27,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:27,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:27,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:29,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:31,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:32,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:32,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:33,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:35,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:37,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:37,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:41:38,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:39,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:41,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:42,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:43,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:43,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:45,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:47,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:47,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:49,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:49,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:51,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:51,305 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:41:52,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:41:53,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:54,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:41:55,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:57,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:41:57,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:41:59,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:00,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:01,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:02,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:03,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:05,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:05,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:07,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:07,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:42:09,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:11,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:11,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:12,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:13,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:15,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:16,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:17,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:17,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:19,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:21,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:21,307 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:42:22,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:22,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:42:23,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:25,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:27,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:27,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:27,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:29,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:31,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:32,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:33,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:33,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:35,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:37,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:37,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:42:38,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:39,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:41,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:42,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:43,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:44,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:45,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:47,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:47,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:49,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:50,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:51,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:51,310 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:42:52,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:42:53,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:55,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:55,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:42:57,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:42:57,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:42:59,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:01,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:01,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:02,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:03,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:05,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:06,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:07,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:07,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:43:09,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:11,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:12,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:12,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:13,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:15,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:17,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:17,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:17,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:19,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:21,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:21,314 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:43:22,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:43:23,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:23,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:25,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:27,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:27,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:28,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:29,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:31,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:32,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:33,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:34,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:35,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:37,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:37,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:43:39,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:39,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:41,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:42,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:43,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:45,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:45,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:47,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:47,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:49,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:50,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:51,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:51,316 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:43:52,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:43:52,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:53,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:53,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:53,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:53,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 04:43:55,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:56,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:43:57,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:43:57,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:43:59,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:01,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:02,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:02,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:03,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:05,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:07,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:07,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:44:07,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:09,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:11,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:12,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:13,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:13,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:15,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:17,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:17,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:18,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:19,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:21,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:21,319 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:44:22,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:44:23,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:24,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:25,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:27,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:27,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:29,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:29,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:31,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:32,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:33,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:35,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:35,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:37,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:37,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:44:39,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:40,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:41,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:42,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:43,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:45,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:46,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:47,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:47,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:49,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:51,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:51,322 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:44:52,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:52,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:44:53,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:55,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:57,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:44:57,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:44:57,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:44:59,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:01,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:02,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:03,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:03,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:05,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:07,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:07,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:45:08,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:09,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:11,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:12,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:13,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:14,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:15,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:17,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:17,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:19,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:19,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:21,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:21,324 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:45:22,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:45:23,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:25,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:25,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:27,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:27,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:29,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:30,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:31,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:32,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:33,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:35,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:36,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:37,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:37,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:45:39,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:41,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:41,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:42,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:43,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:45,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:47,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:47,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:47,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:49,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:51,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:51,327 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:45:52,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:45:52,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:53,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:55,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:57,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:45:57,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:45:58,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:45:59,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:01,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:02,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:03,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:03,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:05,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:07,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:07,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:46:09,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:09,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:11,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:13,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:13,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:14,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:15,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:17,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:18,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:19,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:20,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:21,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:21,330 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:46:22,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:46:23,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:25,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:25,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:27,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:28,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:29,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:31,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:31,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:33,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:33,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:35,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:37,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:37,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:37,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:46:39,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:41,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:42,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:43,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:43,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:45,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:47,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:48,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:48,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:49,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:51,336 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:46:51,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:52,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:46:53,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:53,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:55,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:57,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:46:58,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:46:59,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:46:59,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:01,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:03,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:03,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:04,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:05,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:07,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:07,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:47:09,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:10,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:11,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:13,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:13,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:15,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:15,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:17,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:18,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:19,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:21,335 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:47:21,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:21,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:22,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:47:23,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:25,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:26,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:27,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:28,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:29,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:31,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:32,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:33,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:33,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:35,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:37,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:37,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:47:38,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:39,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:41,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:43,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:43,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:43,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:45,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:47,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:48,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:49,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:49,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:51,338 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:47:51,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:52,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:47:53,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:54,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:47:55,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:57,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:47:58,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:47:59,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:00,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:01,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:03,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:03,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:05,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:05,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:07,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:07,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:48:09,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:11,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:11,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:13,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:13,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:15,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:16,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:17,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:18,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:19,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:21,341 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:48:21,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:22,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:22,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:48:23,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:25,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:27,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:27,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:28,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:29,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:31,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:33,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:33,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:33,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:35,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:37,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:37,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:48:38,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:39,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:41,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:43,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:43,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:44,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:45,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:47,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:48,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:49,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:49,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:51,346 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:48:51,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:52,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:48:53,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:55,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:55,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:48:57,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:48:58,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:48:59,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:01,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:01,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:03,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:03,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:05,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:06,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:07,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:49:07,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:09,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:11,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:12,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:13,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:13,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:15,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:17,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:17,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:18,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:19,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:21,346 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:49:21,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:22,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:49:23,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:23,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:25,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:27,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:28,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:28,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:29,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:31,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:33,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:33,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:34,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:35,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:37,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:49:37,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:39,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:39,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:41,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:43,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:43,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:45,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:45,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:47,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:48,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:49,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:50,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:51,349 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:49:51,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:52,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:49:53,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:55,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:56,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:49:57,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:49:58,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:49:59,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:01,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:01,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:03,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:03,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:05,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:07,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:50:07,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:07,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:09,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:11,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:13,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:13,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:13,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:15,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:17,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:18,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:18,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:19,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:21,352 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:50:21,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:22,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:50:23,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:24,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:25,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:27,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:28,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:29,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:29,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:31,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:33,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:33,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:35,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:35,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:37,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:50:37,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:39,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:40,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:41,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:43,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:43,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:45,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:46,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:47,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:48,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:49,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:51,356 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:50:51,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:51,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:52,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:50:53,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:55,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:57,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:50:57,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:50:58,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:50:59,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:01,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:02,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:03,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:03,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:05,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:07,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:51:07,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:08,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:09,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:11,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:13,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:13,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:13,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:15,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:17,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:18,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:19,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:19,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:21,358 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:51:21,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:22,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:51:23,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:24,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:25,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:27,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:28,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:29,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:30,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:31,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:33,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:33,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:35,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:36,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:37,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:51:37,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:39,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:41,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:41,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:43,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:43,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:45,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:47,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:47,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:48,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:49,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:51,362 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:51:51,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:52,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:51:52,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:53,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:55,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:57,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:51:58,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:51:58,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:51:59,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:01,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:03,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:03,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:03,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:05,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:07,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:52:07,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:09,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:09,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:11,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:13,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:13,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:14,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:15,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:17,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:18,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:19,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:20,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:21,365 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:52:21,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:22,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:52:23,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:25,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:25,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:27,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:28,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:29,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:31,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:31,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:33,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:33,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:35,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:37,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:37,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:52:37,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:39,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:41,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:42,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:43,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:43,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:45,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:47,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:48,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:48,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:49,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:51,371 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:52:51,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:52,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:52:53,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:53,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:55,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:57,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:52:58,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:52:59,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:52:59,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:01,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:03,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:03,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:04,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:05,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:07,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:53:07,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:09,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:10,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:11,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:13,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:13,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:15,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:15,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:17,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:18,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:19,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:21,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:21,372 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:53:21,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:22,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:53:23,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:25,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:26,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:27,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:28,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:29,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:31,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:32,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:33,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:33,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:35,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:37,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:53:37,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:37,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:39,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:41,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:43,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:43,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:43,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:45,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:47,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:48,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:49,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:49,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:51,374 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:53:51,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:52,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:53:53,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:54,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:53:55,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:57,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:53:58,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:53:59,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:00,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:01,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:03,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:03,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:05,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:05,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:07,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:54:07,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:09,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:11,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:11,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:13,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:13,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:15,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:16,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:17,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:18,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:19,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:21,378 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:54:21,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:22,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:22,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:54:23,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:25,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:27,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:27,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:28,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:29,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:31,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:33,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:33,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:33,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:35,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:37,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:54:37,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:38,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:39,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:41,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:43,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:43,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:44,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:45,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:47,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:48,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:49,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:49,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:51,380 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:54:51,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:52,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:54:53,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:55,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:54:55,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:57,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:54:58,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:54:59,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:00,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:01,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:03,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:03,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:05,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:06,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:07,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:55:07,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:09,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:11,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:12,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:13,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:13,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:15,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:17,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:17,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:18,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:19,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:21,383 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:55:21,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:22,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:55:23,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:23,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:25,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:27,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:28,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:28,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:29,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:31,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:33,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:33,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:34,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:35,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:37,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:55:37,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:39,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:39,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:41,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:43,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:43,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:45,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:45,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:47,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:48,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:49,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:50,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:51,386 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:55:51,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:52,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:55:53,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:55,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:56,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:55:57,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:55:58,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:55:59,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:01,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:01,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:03,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:03,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:05,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:07,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:07,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:56:07,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:09,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:11,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:12,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:13,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:13,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:15,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:17,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:18,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:18,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:19,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:21,388 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:56:21,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:22,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:56:23,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:24,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:25,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:27,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:28,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:29,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:29,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:31,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:33,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:33,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:35,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:35,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:37,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:56:37,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:39,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:40,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:41,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:43,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:43,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:45,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:46,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:47,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:48,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:49,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:51,391 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:56:51,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:51,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:52,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:56:53,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:55,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:57,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:56:57,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:56:58,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:56:59,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:01,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:02,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:03,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:03,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:05,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:07,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:57:07,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:08,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:09,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:11,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:13,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:13,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:13,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:15,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:17,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:18,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:19,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:19,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:21,393 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:57:21,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:22,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:57:23,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:24,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:25,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:27,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:28,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:29,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:30,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:31,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:33,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:33,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:35,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:35,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:37,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:57:37,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:39,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:41,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:41,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:43,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:43,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:45,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:47,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:47,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:48,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:49,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:51,396 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:57:51,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:52,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:57:52,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:53,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:55,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:57,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:57:58,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:57:58,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:57:59,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:01,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:03,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:03,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:03,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:05,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:07,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:58:07,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:09,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:10,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:12,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:13,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:14,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:14,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:16,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:18,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:18,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:20,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:20,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:21,402 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:58:22,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:22,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:58:24,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:25,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:26,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:28,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:28,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:30,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:31,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:32,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:33,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:34,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:36,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:36,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:37,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:58:38,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:40,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:42,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:42,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:43,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:44,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:46,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:47,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:48,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:48,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:50,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:51,402 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:58:52,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:52,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:58:53,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:58:54,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:56,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:58,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:58:58,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:58:58,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:00,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:02,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:03,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:04,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:04,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:06,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:07,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:59:08,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:10,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:10,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:12,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:13,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:14,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:15,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:16,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:18,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:18,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:20,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:21,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:21,405 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:59:22,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:22,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:59:24,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:26,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:26,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:28,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:28,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:30,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:32,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:32,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:33,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:34,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:36,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:37,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:59:37,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:38,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:40,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:42,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:43,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:43,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:44,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:46,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:48,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:48,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:48,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:50,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:51,407 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 04:59:52,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:52,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 04:59:54,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:54,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 04:59:56,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:58,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 04:59:58,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 04:59:59,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:00,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:02,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:03,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:04,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:05,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:06,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:07,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:00:08,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:10,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:10,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:12,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:13,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:14,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:16,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:16,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:18,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:18,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:20,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:21,413 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:00:22,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:22,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:22,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:00:24,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:26,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:27,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:28,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:28,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:30,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:32,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:33,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:33,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:34,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:36,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:37,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:00:38,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:38,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:40,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:42,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:43,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:44,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:44,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:46,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:48,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:48,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:49,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:50,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:51,413 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:00:52,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:52,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:00:54,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:55,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:00:56,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:00:58,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:00:58,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:00,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:00,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:02,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:03,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:04,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:06,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:06,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:07,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:01:08,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:10,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:11,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:12,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:13,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:14,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:16,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:17,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:18,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:18,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:20,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:21,416 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:01:22,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:22,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:01:22,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:24,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:26,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:28,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:28,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:28,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:30,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:32,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:33,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:33,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:34,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:36,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:37,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:01:38,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:39,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:40,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:42,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:43,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:44,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:45,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:46,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:48,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:48,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:50,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:50,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:51,419 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:01:52,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:52,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:01:54,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:56,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:01:56,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:01:58,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:01:58,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:00,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:01,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:02,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:03,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:04,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:06,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:07,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:07,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:02:08,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:10,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:12,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:12,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:13,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:14,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:16,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:18,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:18,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:18,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:20,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:21,461 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:02:22,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:22,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:02:23,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:24,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:26,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:28,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:28,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:29,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:30,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:32,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:33,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:34,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:34,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:36,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:37,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:02:38,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:40,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:40,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:42,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:43,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:44,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:46,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:46,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:48,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:48,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:50,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:51,425 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:02:51,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:52,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:52,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:02:54,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:56,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:02:57,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:02:58,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:02:58,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:00,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:02,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:02,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:03,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:04,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:06,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:07,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:03:08,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:08,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:10,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:12,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:13,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:13,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:14,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:16,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:18,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:18,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:19,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:20,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:21,428 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:03:22,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:22,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:03:24,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:24,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:26,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:28,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:28,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:30,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:30,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:32,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:33,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:34,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:35,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:36,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:37,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:03:38,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:40,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:41,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:42,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:43,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:44,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:46,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:46,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:48,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:48,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:50,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:51,430 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:03:52,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:52,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:52,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:03:54,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:56,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:03:57,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:03:58,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:03:58,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:00,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:02,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:03,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:03,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:04,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:06,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:07,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:04:08,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:09,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:10,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:12,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:13,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:14,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:14,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:16,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:18,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:18,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:20,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:20,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:21,433 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:04:22,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:22,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:04:24,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:25,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:26,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:28,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:28,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:30,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:31,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:32,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:33,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:34,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:36,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:36,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:37,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:04:38,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:40,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:42,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:42,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:43,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:44,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:46,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:47,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:48,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:48,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:50,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:51,436 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:04:52,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:52,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:04:53,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:04:54,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:56,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:58,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:04:58,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:04:58,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:00,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:02,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:03,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:04,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:04,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:06,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:07,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:05:08,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:10,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:10,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:12,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:13,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:14,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:15,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:16,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:18,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:18,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:20,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:21,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:21,438 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:05:22,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:22,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:05:24,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:26,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:26,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:28,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:28,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:30,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:32,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:32,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:33,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:34,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:36,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:37,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:05:37,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:38,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:40,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:42,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:43,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:43,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:44,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:46,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:48,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:48,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:48,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:50,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:51,441 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:05:52,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:52,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:05:54,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:05:54,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:56,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:58,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:05:58,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:05:59,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:00,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:02,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:03,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:04,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:05,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:06,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:07,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:06:08,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:10,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:10,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:12,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:13,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:14,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:16,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:16,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:18,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:18,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:20,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:21,444 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:06:22,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:22,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:06:22,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:24,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:26,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:27,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:28,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:28,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:30,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:32,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:33,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:33,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:34,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:36,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:37,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:06:38,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:38,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:40,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:42,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:43,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:44,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:44,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:46,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:48,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:48,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:49,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:50,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:51,447 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:06:52,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:06:52,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:54,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:55,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:06:56,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:06:56,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:56,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:57,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:06:58,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:06:58,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:00,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:00,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:02,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:03,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:04,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:06,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:06,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:07,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:07:08,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:10,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:11,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:12,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:13,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:14,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:16,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:17,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:18,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:18,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:20,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:21,450 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:07:22,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:07:22,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:23,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:24,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:26,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:28,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:28,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:28,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:30,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:32,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:33,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:34,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:34,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:36,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:37,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:07:38,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:39,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:40,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:42,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:43,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:44,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:45,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:46,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:48,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:48,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:50,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:50,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:51,454 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:07:52,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:07:52,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:54,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:56,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:07:56,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:07:58,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:07:58,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:00,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:01,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:02,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:03,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:04,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:06,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:07,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:07,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:08:08,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:10,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:12,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:12,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:13,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:14,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:16,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:18,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:18,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:18,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:20,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:21,456 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:08:22,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:08:22,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:23,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:24,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:26,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:28,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:28,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:29,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:30,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:32,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:33,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:34,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:35,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:36,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:37,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:08:38,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:40,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:40,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:42,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:43,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:44,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:46,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:46,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:48,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:48,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:50,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:51,458 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:08:51,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:52,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:08:52,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:54,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:56,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:08:57,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:08:58,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:08:58,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:00,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:02,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:02,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:03,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:04,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:06,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:07,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:09:08,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:08,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:10,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:12,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:13,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:13,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:14,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:16,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:18,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:18,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:19,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:20,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:21,461 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:09:22,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:09:22,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:24,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:24,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:26,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:28,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:28,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:30,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:30,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:32,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:33,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:34,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:35,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:36,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:37,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:09:38,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:40,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:41,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:42,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:43,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:44,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:46,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:46,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:48,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:48,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:50,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:51,468 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:09:52,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:52,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:09:52,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:54,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:56,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:09:58,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:09:58,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:09:58,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:00,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:02,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:03,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:03,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:04,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:06,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:07,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:10:08,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:09,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:10,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:12,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:13,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:14,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:14,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:16,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:18,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:18,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:20,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:20,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:21,466 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:10:22,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:10:22,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:24,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:25,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:26,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:28,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:28,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:30,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:31,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:32,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:33,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:34,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:36,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:36,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:37,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:10:38,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:40,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:42,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:42,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:43,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:44,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:46,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:47,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:48,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:48,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:50,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:51,469 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:10:52,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:10:52,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:53,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:10:54,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:56,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:58,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:10:58,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:10:58,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:00,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:02,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:03,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:04,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:04,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:06,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:07,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:11:08,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:09,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:10,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:12,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:13,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:14,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:15,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:16,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:18,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:18,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:20,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:21,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:21,472 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:11:22,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:11:22,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:24,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:26,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:26,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:28,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:28,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:30,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:32,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:32,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:33,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:34,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:36,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:37,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:11:37,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:38,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:40,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:42,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:43,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:43,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:44,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:46,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:48,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:48,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:48,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:50,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:51,475 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:11:52,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:11:52,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:54,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:11:54,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:56,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:58,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:11:58,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:11:59,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:00,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:02,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:03,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:04,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:05,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:06,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:07,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:12:08,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:10,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:10,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:12,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:13,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:14,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:16,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:16,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:18,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:18,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:20,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:21,478 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:12:21,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:22,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:12:22,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:24,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:26,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:26,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:28,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:28,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:30,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:31,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:32,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:33,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:34,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:36,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:37,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:37,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:12:38,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:40,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:42,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:42,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:43,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:44,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:46,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:48,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:48,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:48,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:50,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:51,481 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:12:52,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:12:52,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:53,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:12:54,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:56,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:58,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:12:58,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:12:59,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:00,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:02,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:03,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:04,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:04,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:06,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:07,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:13:08,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:10,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:10,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:12,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:13,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:14,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:15,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:16,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:18,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:18,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:20,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:21,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:21,483 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:13:22,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:13:22,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:24,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:26,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:27,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:28,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:28,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:30,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:32,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:32,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:33,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:34,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:36,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:37,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:13:38,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:38,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:40,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:42,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:43,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:43,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:44,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:46,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:48,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:48,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:49,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:50,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:51,486 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:13:52,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:13:52,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:54,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:13:54,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:56,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:13:58,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:13:58,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:00,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:00,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:02,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:03,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:04,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:05,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:06,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:07,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:14:08,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:10,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:11,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:12,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:13,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:14,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:16,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:16,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:18,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:18,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:20,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:21,489 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:14:22,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:14:22,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:22,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:24,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:26,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:27,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:28,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:28,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:30,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:32,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:33,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:33,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:34,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:36,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:37,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:14:38,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:39,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:40,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:42,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:43,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:44,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:44,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:46,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:48,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:48,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:50,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:50,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:51,493 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:14:52,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:14:52,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:54,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:55,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:14:56,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:14:58,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:14:58,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:00,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:01,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:02,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:03,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:04,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:06,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:06,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:07,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:15:08,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:11,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:12,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:13,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:13,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:15,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:17,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:17,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:18,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:19,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:21,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:21,497 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:15:22,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:15:23,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:23,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:25,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:27,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:28,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:28,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:29,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:31,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:33,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:33,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:34,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:35,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:37,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:37,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:15:39,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:39,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:41,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:43,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:43,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:45,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:45,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:47,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:48,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:49,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:50,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:51,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:51,498 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:15:52,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:15:53,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:55,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:56,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:15:57,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:15:58,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:15:59,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:01,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:02,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:03,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:03,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:05,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:07,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:07,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:16:07,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:09,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:11,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:13,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:13,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:13,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:15,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:17,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:18,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:18,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:19,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:21,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:21,501 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:16:22,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:16:23,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:24,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:25,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:27,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:28,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:29,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:29,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:31,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:33,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:33,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:35,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:35,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:37,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:37,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:16:39,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:40,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:41,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:43,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:43,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:45,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:46,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:47,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:48,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:49,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:51,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:51,503 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:16:51,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:52,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:16:53,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:55,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:57,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:16:57,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:16:58,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:16:59,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:01,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:02,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:03,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:03,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:05,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:07,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:07,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:17:08,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:09,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:11,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:13,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:13,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:14,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:15,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:17,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:18,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:19,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:19,604 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:21,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:21,509 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:17:22,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:17:23,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:25,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:25,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:27,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:28,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:29,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:30,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:31,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:33,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:33,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:35,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:36,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:37,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:37,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:17:39,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:41,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:41,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:43,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:43,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:45,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:47,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:47,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:48,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:49,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:51,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:51,509 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:17:52,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:17:52,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:53,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:17:57,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:17:58,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:17:59,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:01,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:03,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:03,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:03,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:05,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:07,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:07,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:18:08,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:09,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:11,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:13,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:13,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:14,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:15,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:17,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:18,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:19,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:19,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:21,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:21,512 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:18:22,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:18:23,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:25,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:25,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:27,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:28,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:29,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:30,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:31,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:33,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:33,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:35,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:36,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:37,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:37,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:18:39,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:41,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:41,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:43,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:43,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:45,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:47,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:47,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:48,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:49,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:51,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:51,514 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:18:52,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:18:52,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:53,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:55,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:57,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:18:58,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:18:58,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:18:59,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:01,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:03,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:03,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:03,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:05,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:07,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:07,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:19:09,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:09,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:11,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:13,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:13,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:14,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:15,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:17,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:18,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:19,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:20,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:21,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:21,516 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:19:22,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:19:23,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:25,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:26,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:27,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:28,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:29,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:31,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:31,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:33,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:33,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:35,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:37,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:37,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:37,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:19:39,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:41,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:42,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:43,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:43,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:45,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:47,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:48,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:48,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:49,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:51,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:51,519 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:19:52,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:19:53,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:53,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:55,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:57,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:19:58,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:19:58,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:19:59,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:01,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:03,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:03,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:03,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:05,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:07,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:07,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:20:09,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:09,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:11,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:13,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:13,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:14,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:15,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:17,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:18,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:19,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:19,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:21,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:21,522 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:20:22,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:20:23,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:24,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:25,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:27,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:28,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:29,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:30,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:31,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:33,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:33,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:35,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:35,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:37,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:20:40,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:43,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:43,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:20:43,989 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 05:20:43,996 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 05:20:44,006 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 05:20:44,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 05:20:45,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:46,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:49,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:20:51,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:51,525 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:20:53,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:20:56,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:20:57,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:20:58,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:01,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:03,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:07,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:07,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:07,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:21:08,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:21:08,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:21:08,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 05:21:08,852 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 05:21:08,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:21:09,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:09,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_11_aa86ab67448a943ea3cd.table.json 2025-09-18 05:21:09,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 05:21:09,706 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmp3sa8wrbh 2025-09-18 05:21:09,783 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/99lb0e18-media/table/rollouts_11_aa86ab67448a943ea3cd.table.json 2025-09-18 05:21:10,431 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MjUwOTA4Mw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MjM1ODMwOQ==', 'versionIndex': 9}}} 2025-09-18 05:21:10,431 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 05:21:10,432 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_11_aa86ab67448a943ea3cd.table.json with policy now 2025-09-18 05:21:10,734 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/4g96agr3-media/table/rollouts_11_aa86ab67448a943ea3cd.table.json 2025-09-18 05:21:11,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:12,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:13,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:13,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:15,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:17,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:17,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:18,054 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:19,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:21,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:21,552 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:21:22,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:21:23,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:23,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:25,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:27,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:28,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:28,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:29,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:31,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:33,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:33,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:34,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:35,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:37,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:37,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:21:39,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:39,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:41,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:43,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:43,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:45,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:45,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:47,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:48,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:49,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:50,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:51,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:51,555 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:21:52,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:21:53,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:55,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:56,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:21:57,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:21:58,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:21:59,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:01,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:01,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:03,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:03,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:05,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:07,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:07,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:07,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:22:09,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:11,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:13,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:13,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:13,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:15,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:17,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:18,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:18,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:19,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:21,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:21,557 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:22:22,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:22:23,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:24,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:25,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:27,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:28,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:29,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:29,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:31,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:33,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:33,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:35,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:35,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:37,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:37,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:22:39,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:40,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:41,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:43,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:43,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:45,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:46,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:47,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:48,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:49,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:51,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:51,560 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:22:51,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:52,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:22:53,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:55,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:57,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:22:57,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:22:58,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:22:59,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:01,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:02,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:03,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:03,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:05,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:07,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:07,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:23:08,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:09,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:11,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:13,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:13,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:13,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:15,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:17,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:18,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:19,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:19,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:21,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:21,564 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:23:22,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:23:23,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:24,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:25,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:27,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:28,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:29,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:30,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:31,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:33,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:33,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:35,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:36,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:37,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:37,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:23:39,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:41,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:41,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:43,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:43,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:45,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:47,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:47,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:48,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:49,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:51,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:51,567 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:23:52,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:52,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:23:53,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:55,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:57,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:23:58,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:23:58,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:23:59,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:01,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:03,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:03,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:03,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:05,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:07,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:07,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:24:09,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:09,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:11,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:13,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:13,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:14,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:15,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:17,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:18,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:19,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:20,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:21,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:21,570 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:24:22,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:24:23,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:25,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:25,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:27,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:28,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:29,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:31,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:31,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:33,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:33,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:35,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:36,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:37,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:37,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:24:39,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:41,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:42,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:43,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:43,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:45,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:47,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:48,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:48,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:49,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:51,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:51,572 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:24:52,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:24:53,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:53,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:55,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:57,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:24:58,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:24:59,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:24:59,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:01,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:03,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:03,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:04,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:05,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:07,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:07,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:25:09,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:10,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:11,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:13,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:13,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:15,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:15,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:17,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:18,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:19,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:21,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:21,578 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:25:21,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:22,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:25:23,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:25,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:26,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:27,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:28,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:29,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:31,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:32,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:33,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:33,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:35,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:37,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:37,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:25:37,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:39,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:41,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:43,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:43,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:43,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:45,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:47,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:48,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:48,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:49,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:51,578 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:25:51,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:52,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:25:53,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:54,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:25:55,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:57,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:58,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:25:59,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:25:59,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:01,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:03,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:03,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:05,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:05,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:07,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:07,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:26:09,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:11,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:11,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:13,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:13,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:15,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:16,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:17,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:18,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:19,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:21,580 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:26:21,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:21,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:22,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:26:23,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:25,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:26,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:27,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:28,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:29,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:31,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:31,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:33,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:33,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:35,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:37,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:37,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:37,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:26:39,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:41,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:42,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:43,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:43,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:45,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:47,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:48,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:48,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:49,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:51,584 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:26:51,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:52,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:26:53,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:53,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:55,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:57,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:26:58,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:26:59,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:26:59,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:01,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:03,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:03,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:04,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:05,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:07,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:07,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:27:09,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:10,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:11,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:13,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:13,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:15,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:16,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:17,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:18,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:19,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:21,587 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:27:21,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:21,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:22,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:27:23,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:25,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:27,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:27,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:28,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:29,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:31,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:32,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:33,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:33,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:35,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:37,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:37,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:27:38,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:39,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:41,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:43,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:43,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:43,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:45,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:47,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:48,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:49,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:49,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:51,589 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:27:51,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:52,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:27:53,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:54,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:27:55,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:57,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:27:58,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:27:59,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:00,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:01,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:03,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:03,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:05,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:05,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:07,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:07,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:28:09,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:11,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:11,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:13,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:13,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:15,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:16,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:17,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:18,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:19,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:21,592 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:28:21,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:22,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:22,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:28:23,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:25,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:27,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:27,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:28,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:29,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:31,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:33,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:33,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:33,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:35,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:37,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:37,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:28:39,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:39,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:41,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:43,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:43,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:44,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:45,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:47,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:48,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:49,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:50,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:51,594 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:28:51,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:52,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:28:53,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:55,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:28:55,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:57,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:28:58,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:28:59,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:01,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:01,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:03,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:03,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:05,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:06,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:07,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:07,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:29:09,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:11,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:12,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:13,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:13,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:15,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:17,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:17,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:18,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:19,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:21,599 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:29:21,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:22,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:29:23,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:23,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:25,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:27,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:28,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:28,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:29,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:31,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:33,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:33,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:34,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:35,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:37,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:29:37,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:39,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:40,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:41,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:43,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:43,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:45,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:45,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:47,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:48,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:49,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:51,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:51,600 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:29:51,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:52,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:29:53,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:55,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:56,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:29:57,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:29:58,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:29:59,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:01,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:02,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:03,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:03,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:05,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:07,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:07,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:30:07,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:09,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:11,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:13,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:13,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:13,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:15,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:17,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:18,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:18,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:19,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:21,603 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:30:21,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:22,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:30:23,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:24,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:25,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:27,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:28,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:29,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:29,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:31,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:33,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:33,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:35,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:35,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:37,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:30:37,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:39,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:40,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:41,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:43,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:43,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:45,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:46,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:47,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:48,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:49,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:51,605 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:30:51,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:51,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:52,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:30:53,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:55,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:57,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:30:57,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:30:58,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:30:59,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:01,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:03,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:03,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:03,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:05,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:07,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:31:07,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:08,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:09,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:11,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:13,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:13,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:14,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:15,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:17,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:18,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:19,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:19,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:21,610 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:31:21,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:22,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:31:22,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:22,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:31:23,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:25,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:25,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:27,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:28,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:29,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:30,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:31,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:33,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:33,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:35,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:36,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:37,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:31:37,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:39,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:41,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:42,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:43,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:43,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:45,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:47,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:47,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:48,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:49,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:51,611 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:31:51,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:52,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:31:53,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:53,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:55,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:57,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:31:58,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:31:58,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:31:59,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:01,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:03,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:03,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:04,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:05,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:07,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:32:07,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:09,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:09,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:11,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:13,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:13,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:15,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:15,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:17,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:18,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:19,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:20,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:21,615 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:32:22,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:22,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:32:24,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:26,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:26,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:28,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:28,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:30,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:31,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:32,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:33,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:34,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:36,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:37,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:37,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:32:38,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:40,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:42,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:42,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:43,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:44,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:46,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:48,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:48,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:48,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:50,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:51,619 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:32:52,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:52,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:32:53,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:32:54,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:56,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:58,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:32:58,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:32:59,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:00,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:02,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:03,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:04,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:05,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:06,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:07,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:33:08,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:10,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:10,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:12,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:13,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:14,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:16,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:16,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:18,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:18,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:20,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:21,623 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:33:21,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:22,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:22,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:33:24,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:26,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:27,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:28,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:28,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:30,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:32,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:32,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:33,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:34,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:36,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:37,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:33:38,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:38,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:40,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:42,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:43,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:43,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:44,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:46,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:48,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:48,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:33:49,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:50,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:51,625 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:33:52,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:52,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:33:54,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:54,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:33:56,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:58,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:33:58,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:00,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:00,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:02,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:03,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:04,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:05,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:06,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:07,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:34:08,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:10,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:11,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:12,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:13,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:14,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:16,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:16,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:18,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:18,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:20,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:21,628 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:34:22,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:22,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:22,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:34:24,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:26,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:28,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:28,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:28,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:30,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:32,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:33,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:33,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:34,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:36,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:37,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:34:38,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:39,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:40,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:42,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:43,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:44,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:44,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:46,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:48,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:48,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:34:50,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:50,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:51,631 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:34:52,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:52,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:34:54,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:55,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:34:56,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:58,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:34:58,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:00,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:01,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:02,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:03,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:04,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:06,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:06,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:07,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:35:08,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:10,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:12,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:12,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:13,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:14,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:16,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:17,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:18,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:18,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:20,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:21,633 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:35:22,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:22,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:35:23,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:24,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:26,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:28,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:28,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:28,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:30,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:32,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:33,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:34,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:34,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:36,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:37,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:35:38,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:40,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:40,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:42,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:43,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:44,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:45,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:46,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:48,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:48,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:50,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:51,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:51,636 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:35:52,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:52,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:35:54,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:56,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:35:56,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:35:58,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:35:58,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:00,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:02,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:02,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:03,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:04,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:06,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:07,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:07,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:36:08,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:10,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:12,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:13,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:13,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:14,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:16,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:18,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:18,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:18,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:20,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:21,639 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:36:22,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:22,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:36:24,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:24,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:26,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:28,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:28,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:29,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:30,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:32,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:33,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:34,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:35,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:36,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:37,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:36:38,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:40,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:40,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:42,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:43,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:44,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:46,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:46,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:48,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:48,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:50,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:51,642 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:36:51,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:52,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:52,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:36:54,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:56,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:36:57,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:36:58,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:36:58,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:00,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:02,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:03,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:03,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:04,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:06,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:07,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:37:08,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:08,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:10,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:12,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:13,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:14,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:14,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:16,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:18,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:18,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:19,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:20,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:21,645 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:37:22,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:22,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:37:24,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:25,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:26,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:28,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:28,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:30,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:30,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:32,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:33,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:34,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:36,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:36,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:37,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:37:38,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:40,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:41,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:42,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:43,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:44,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:46,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:47,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:48,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:48,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:50,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:51,647 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:37:52,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:52,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:37:52,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:37:54,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:56,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:58,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:37:58,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:37:58,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:00,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:02,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:03,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:03,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:04,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:06,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:07,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:38:08,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:09,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:10,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:12,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:13,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:14,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:14,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:16,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:18,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:18,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:20,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:20,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:21,650 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:38:22,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:22,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:38:24,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:26,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:26,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:28,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:28,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:30,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:31,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:32,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:33,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:34,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:36,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:37,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:37,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:38:38,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:40,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:42,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:42,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:43,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:44,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:46,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:48,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:48,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:48,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:50,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:51,655 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:38:52,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:52,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:38:53,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:38:54,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:56,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:58,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:38:58,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:38:59,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:00,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:02,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:03,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:04,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:04,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:06,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:07,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:39:08,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:10,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:10,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:12,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:13,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:14,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:15,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:16,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:18,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:18,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:20,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:21,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:21,656 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:39:22,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:22,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:39:24,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:26,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:26,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:28,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:28,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:30,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:32,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:32,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:33,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:34,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:36,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:37,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:39:38,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:38,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:40,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:42,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:43,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:43,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:44,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:46,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:48,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:48,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:49,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:50,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:51,659 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:39:52,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:52,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:39:54,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:54,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:39:56,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:39:58,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:39:58,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:00,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:00,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:02,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:03,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:04,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:05,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:06,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:07,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:40:08,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:10,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:11,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:12,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:13,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:14,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:16,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:16,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:18,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:18,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:20,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:21,662 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:40:22,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:22,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:22,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:40:24,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:26,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:27,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:28,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:28,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:30,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:32,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:33,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:33,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:34,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:36,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:37,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:40:38,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:38,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:40,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:42,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:43,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:44,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:44,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:46,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:48,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:48,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:49,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:50,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:51,666 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:40:52,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:52,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:40:54,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:55,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:40:56,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:40:58,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:40:58,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:00,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:01,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:02,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:03,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:04,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:06,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:06,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:07,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:41:08,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:10,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:12,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:12,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:13,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:14,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:16,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:17,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:18,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:18,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:20,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:21,667 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:41:22,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:22,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:41:23,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:24,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:26,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:28,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:28,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:28,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:30,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:32,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:33,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:34,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:34,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:36,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:37,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:41:38,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:39,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:40,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:42,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:43,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:44,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:45,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:46,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:48,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:48,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:50,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:50,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:51,669 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:41:52,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:52,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:41:54,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:56,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:41:56,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:41:58,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:41:58,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:00,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:01,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:02,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:03,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:04,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:06,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:07,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:07,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:42:08,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:10,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:12,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:13,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:13,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:14,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:16,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:18,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:18,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:18,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:20,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:21,672 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:42:22,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:22,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:42:24,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:24,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:26,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:28,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:28,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:29,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:30,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:32,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:33,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:34,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:35,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:36,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:37,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:42:38,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:40,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:40,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:42,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:43,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:44,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:46,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:46,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:48,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:48,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:50,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:51,676 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:42:51,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:52,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:52,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:42:54,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:56,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:42:57,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:42:58,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:42:58,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:00,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:02,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:02,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:03,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:04,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:06,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:07,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:43:08,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:08,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:10,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:12,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:13,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:13,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:14,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:16,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:18,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:18,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:19,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:20,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:21,677 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:43:22,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:22,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:43:24,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:24,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:26,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:28,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:28,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:30,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:30,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:32,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:33,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:34,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:36,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:36,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:37,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:43:38,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:40,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:41,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:42,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:43,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:44,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:46,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:47,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:48,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:48,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:50,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:51,681 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:43:52,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:52,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:52,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:43:54,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:56,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:43:58,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:43:58,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:43:58,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:00,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:02,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:03,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:03,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:04,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:06,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:07,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:44:08,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:09,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:10,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:12,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:13,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:14,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:14,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:16,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:18,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:18,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:20,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:20,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:21,683 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:44:22,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:22,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:44:24,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:25,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:26,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:28,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:28,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:30,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:31,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:32,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:33,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:34,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:36,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:36,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:37,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:44:38,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:40,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:42,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:42,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:43,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:44,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:46,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:48,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:48,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:48,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:50,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:51,686 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:44:52,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:52,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:44:53,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:44:54,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:56,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:58,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:44:58,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:44:59,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:00,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:02,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:03,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:04,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:04,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:06,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:07,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:45:08,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:10,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:10,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:12,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:13,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:14,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:15,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:16,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:18,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:18,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:20,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:21,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:21,689 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:45:22,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:22,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:45:24,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:26,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:26,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:28,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:28,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:30,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:32,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:32,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:33,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:34,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:36,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:37,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:45:37,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:38,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:40,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:42,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:43,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:43,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:44,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:46,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:48,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:48,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:48,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:50,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:51,692 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:45:52,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:52,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:45:54,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:45:54,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:56,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:58,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:45:58,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:45:59,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:00,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:02,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:03,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:04,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:05,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:06,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:07,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:46:08,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:10,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:11,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:12,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:13,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:14,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:16,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:16,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:18,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:18,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:20,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:21,695 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:46:21,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:22,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:22,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:46:24,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:26,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:27,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:28,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:28,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:30,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:32,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:33,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:33,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:34,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:36,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:37,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:46:38,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:38,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:40,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:42,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:43,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:44,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:44,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:46,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:48,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:48,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:49,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:50,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:51,698 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:46:52,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:46:52,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:54,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:55,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:46:56,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:46:58,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:46:58,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:00,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:00,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:02,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:03,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:04,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:06,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:06,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:07,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:47:08,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:10,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:11,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:12,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:13,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:14,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:16,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:17,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:18,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:18,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:20,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:21,701 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:47:22,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:47:22,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:23,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:24,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:26,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:28,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:28,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:28,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:30,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:32,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:33,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:34,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:34,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:36,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:37,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:47:38,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:39,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:40,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:42,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:43,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:44,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:45,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:46,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:48,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:48,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:50,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:50,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:51,704 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:47:52,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:47:52,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:54,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:56,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:47:56,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:47:58,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:47:58,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:00,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:01,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:02,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:03,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:04,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:06,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:07,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:07,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:48:08,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:10,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:12,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:12,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:13,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:14,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:16,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:18,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:18,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:18,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:20,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:21,709 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:48:22,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:48:22,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:23,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:24,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:26,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:28,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:28,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:29,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:30,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:32,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:33,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:34,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:35,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:36,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:37,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:48:38,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:40,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:40,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:42,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:43,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:44,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:46,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:46,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:48,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:48,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:50,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:51,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:51,709 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:48:52,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:48:52,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:54,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:56,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:48:57,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:48:58,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:48:58,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:00,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:02,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:02,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:03,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:04,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:06,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:07,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:49:08,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:08,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:10,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:12,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:13,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:13,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:14,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:16,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:18,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:18,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:19,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:20,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:21,714 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:49:22,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:49:22,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:24,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:24,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:26,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:28,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:29,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:30,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:31,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:33,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:33,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:35,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:35,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:37,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:37,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:49:39,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:41,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:41,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:43,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:43,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:45,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:46,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:47,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:48,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:49,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:51,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:51,717 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:49:52,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:52,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:49:53,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:55,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:57,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:49:58,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:49:58,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:49:59,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:01,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:03,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:03,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:03,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:05,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:07,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:07,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:50:09,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:09,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:11,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:13,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:13,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:14,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:15,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:17,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:18,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:19,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:20,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:21,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:21,722 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:50:22,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:50:23,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:25,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:25,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:27,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:28,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:29,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:31,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:31,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:33,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:33,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:35,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:36,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:37,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:37,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:50:39,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:41,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:42,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:43,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:43,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:45,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:47,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:47,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:48,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:49,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:51,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:51,722 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:50:52,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:50:53,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:53,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:55,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:57,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:50:58,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:50:58,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:50:59,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:01,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:03,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:03,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:04,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:05,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:07,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:07,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:51:09,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:10,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:11,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:13,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:13,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:15,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:15,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:17,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:18,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:19,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:21,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:21,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:21,725 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:51:22,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:51:23,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:25,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:26,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:27,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:28,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:29,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:31,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:32,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:33,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:33,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:35,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:37,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:37,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:37,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:51:39,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:41,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:43,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:43,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:43,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:45,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:47,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:48,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:48,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:49,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:51,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:51,727 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:51:52,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:51:53,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:54,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:51:55,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:57,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:58,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:51:59,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:51:59,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:01,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:03,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:03,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:05,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:05,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:07,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:07,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:52:09,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:10,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:11,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:13,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:13,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:15,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:16,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:17,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:18,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:19,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:21,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:21,731 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:52:21,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:22,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:52:23,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:25,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:27,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:27,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:28,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:29,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:31,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:33,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:33,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:33,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:35,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:37,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:52:38,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:38,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:39,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:41,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:43,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:43,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:44,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:45,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:47,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:48,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:49,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:49,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:51,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:51,734 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:52:52,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:52:53,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:55,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:55,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:52:57,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:52:58,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:52:59,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:00,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:01,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:03,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:03,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:06,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:06,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:07,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:53:08,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:10,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:11,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:11,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:13,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:13,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:15,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:17,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:18,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:18,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:19,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:21,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:21,736 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:53:22,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:53:22,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:23,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:26,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:28,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:28,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:28,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:30,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:32,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:33,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:34,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:34,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:36,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:37,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:53:38,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:39,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:40,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:42,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:43,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:44,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:45,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:46,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:48,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:48,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:53:50,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:50,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:51,741 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:53:52,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:52,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:53:54,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:56,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:53:56,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:58,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:53:58,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:00,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:01,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:02,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:03,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:04,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:06,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:07,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:07,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:54:08,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:10,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:12,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:12,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:13,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:14,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:16,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:18,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:18,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:18,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:20,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:21,742 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:54:22,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:22,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:54:23,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:24,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:26,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:26,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:26,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:27,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 05:54:28,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:28,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:29,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:30,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:32,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:33,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:34,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:35,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:36,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:37,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:54:38,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:40,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:40,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:42,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:43,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:44,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:46,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:46,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:48,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:48,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:54:50,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:51,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:51,745 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:54:52,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:52,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:54:54,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:56,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:57,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:54:58,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:54:58,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:00,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:02,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:02,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:03,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:04,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:06,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:07,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:55:08,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:08,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:10,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:12,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:13,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:13,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:14,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:16,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:18,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:18,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:19,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:20,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:21,748 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:55:22,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:22,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:55:24,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:24,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:26,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:28,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:28,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:30,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:30,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:32,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:33,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:34,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:35,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:36,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:37,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:55:38,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:40,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:41,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:42,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:43,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:44,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:46,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:46,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:48,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:48,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:55:50,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:51,753 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:55:52,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:52,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:52,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:55:54,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:56,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:58,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:55:58,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:55:58,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:00,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:02,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:03,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:03,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:04,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:06,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:07,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:56:08,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:09,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:10,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:12,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:13,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:14,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:14,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:16,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:18,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:18,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:20,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:20,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:21,754 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:56:22,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:22,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:56:24,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:25,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:26,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:28,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:28,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:30,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:31,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:32,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:33,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:34,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:36,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:36,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:37,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:56:38,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:40,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:42,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:42,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:43,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:44,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:46,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:47,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:48,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:48,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:50,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:51,756 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:56:52,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:52,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:56:53,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:56:54,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:56,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:58,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:56:58,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:56:58,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:00,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:02,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:03,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:04,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:04,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:06,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:07,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:57:08,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:10,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:10,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:12,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:13,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:14,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:15,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:16,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:18,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:18,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:20,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:21,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:21,759 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:57:22,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:22,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:57:24,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:26,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:26,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:28,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:28,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:30,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:32,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:32,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:33,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:34,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:36,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:37,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:37,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:57:38,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:40,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:42,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:43,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:43,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:44,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:46,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:48,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:48,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:48,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:50,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:51,764 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:57:52,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:52,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:57:54,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:57:54,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:56,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:58,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:57:58,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:57:59,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:00,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:02,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:03,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:04,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:05,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:06,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:07,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:58:08,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:10,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:10,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:12,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:13,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:14,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:16,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:16,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:18,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:18,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:20,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:21,765 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:58:21,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:22,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:22,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:58:24,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:26,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:27,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:28,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:28,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:30,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:32,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:33,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:33,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:34,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:36,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:37,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:58:38,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:38,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:40,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:42,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:43,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:44,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:44,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:46,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:48,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:48,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:49,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:50,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:51,768 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:58:52,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:52,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:58:54,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:55,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:58:56,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:58:58,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:58:58,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:00,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:00,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:02,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:03,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:04,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:06,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:06,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:07,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:59:08,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:10,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:11,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:12,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:13,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:14,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:16,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:17,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:18,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:18,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:20,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:21,770 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:59:22,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:22,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:59:22,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:24,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:26,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:28,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:28,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:28,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:30,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:32,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:33,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:33,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:34,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:36,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:37,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:59:38,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:39,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:40,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:42,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:43,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:44,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:45,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:46,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:48,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:48,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:50,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:50,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:51,773 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 05:59:52,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:52,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 05:59:54,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:56,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 05:59:56,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 05:59:58,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 05:59:58,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:00,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:01,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:02,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:03,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:04,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:06,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:06,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:07,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:00:08,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:10,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:12,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:12,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:13,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:14,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:16,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:17,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:18,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:18,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:20,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:21,776 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:00:22,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:22,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:00:22,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:24,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:26,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:28,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:28,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:28,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:30,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:32,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:33,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:33,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:34,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:36,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:37,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:00:38,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:38,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:40,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:42,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:43,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:44,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:44,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:46,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:00:46,232 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 06:00:46,239 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 06:00:46,249 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 06:00:46,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 06:00:46,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:48,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:49,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:49,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:51,779 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:00:52,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:00:54,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:00:58,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:00:58,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:00:59,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:00,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:04,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:05,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:08,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:01:08,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:01:09,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:01:09,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 06:01:09,266 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 06:01:09,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:01:09,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_12_00b26c37b2bb10ac0977.table.json 2025-09-18 06:01:09,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 06:01:10,277 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/m0s34bpc-media/table/rollouts_12_00b26c37b2bb10ac0977.table.json 2025-09-18 06:01:10,309 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmptkqapeoq 2025-09-18 06:01:10,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:11,022 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MjY2MzYzNg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MjUwOTA4Mw==', 'versionIndex': 10}}} 2025-09-18 06:01:11,023 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 06:01:11,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:11,023 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_12_00b26c37b2bb10ac0977.table.json with policy now 2025-09-18 06:01:11,328 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/pus3wwxw-media/table/rollouts_12_00b26c37b2bb10ac0977.table.json 2025-09-18 06:01:12,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:14,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:14,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:16,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:16,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:18,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:19,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:20,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:21,782 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:01:21,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:22,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:23,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:01:24,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:26,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:26,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:28,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:28,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:30,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:31,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:32,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:33,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:34,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:36,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:37,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:38,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:01:38,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:40,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:42,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:43,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:43,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:44,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:46,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:48,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:48,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:48,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:50,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:51,787 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:01:52,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:53,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:01:54,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:01:54,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:56,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:58,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:01:58,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:01:59,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:00,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:02,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:03,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:04,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:05,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:06,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:08,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:02:08,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:10,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:10,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:12,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:13,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:14,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:16,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:16,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:18,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:18,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:20,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:21,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:21,788 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:02:22,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:23,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:02:24,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:26,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:27,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:28,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:28,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:30,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:32,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:32,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:33,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:34,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:36,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:38,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:02:38,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:38,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:40,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:42,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:43,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:43,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:44,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:46,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:48,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:48,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:49,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:50,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:51,791 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:02:52,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:53,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:02:54,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:54,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:02:56,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:02:58,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:02:58,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:00,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:00,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:02,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:03,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:05,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:08,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:03:08,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:10,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:11,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:12,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:13,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:14,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:16,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:17,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:18,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:18,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:20,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:21,794 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:03:22,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:22,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:23,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:03:24,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:26,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:28,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:28,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:28,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:30,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:32,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:33,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:33,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:34,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:36,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:38,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:03:38,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:39,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:40,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:42,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:43,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:44,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:44,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:46,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:48,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:48,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:50,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:50,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:51,799 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:03:52,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:53,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:03:54,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:56,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:03:56,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:03:58,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:03:58,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:00,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:01,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:02,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:03,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:04,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:06,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:07,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:08,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:04:08,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:10,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:12,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:12,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:13,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:14,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:16,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:18,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:18,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:18,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:20,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:21,801 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:04:22,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:23,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:04:23,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:24,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:26,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:28,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:28,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:29,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:30,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:32,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:33,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:34,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:34,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:36,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:38,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:04:38,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:40,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:40,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:42,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:43,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:44,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:45,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:46,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:48,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:48,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:50,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:51,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:51,803 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:04:52,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:53,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:04:54,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:56,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:04:56,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:04:58,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:04:58,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:00,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:02,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:02,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:03,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:04,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:06,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:07,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:08,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:05:08,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:10,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:12,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:13,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:13,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:14,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:16,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:18,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:18,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:19,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:20,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:21,806 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:05:22,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:23,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:05:24,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:24,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:26,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:28,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:28,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:30,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:30,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:32,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:33,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:34,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:35,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:36,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:38,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:05:38,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:40,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:41,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:42,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:43,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:44,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:46,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:46,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:48,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:48,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:50,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:51,811 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:05:51,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:52,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:53,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:05:54,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:56,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:05:57,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:05:58,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:05:59,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:01,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:02,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:03,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:03,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:05,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:07,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:07,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:08,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:06:09,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:11,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:13,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:13,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:13,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:15,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:17,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:18,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:18,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:19,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:21,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:21,813 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:06:23,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:23,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:06:24,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:25,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:27,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:28,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:29,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:29,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:31,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:33,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:33,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:35,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:35,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:37,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:38,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:06:39,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:40,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:41,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:43,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:43,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:45,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:46,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:47,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:48,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:49,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:51,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:51,841 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:06:51,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:53,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:53,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:06:55,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:57,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:06:57,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:06:58,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:06:59,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:01,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:02,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:03,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:03,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:05,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:07,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:08,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:07:08,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:09,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:11,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:13,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:13,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:14,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:15,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:17,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:18,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:19,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:19,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:21,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:21,844 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:07:23,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:07:23,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:25,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:25,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:27,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:28,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:29,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:30,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:31,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:33,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:33,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:35,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:36,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:37,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:38,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:07:39,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:41,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:41,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:43,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:43,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:45,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:47,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:47,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:48,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:49,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:51,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:51,849 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:07:52,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:53,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:07:53,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:55,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:57,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:07:58,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:07:58,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:07:59,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:01,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:03,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:03,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:03,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:05,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:07,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:08,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:08:09,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:09,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:11,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:13,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:13,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:14,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:15,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:17,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:18,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:19,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:20,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:21,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:21,849 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:08:23,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:08:23,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:25,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:25,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:27,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:28,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:29,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:31,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:31,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:33,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:33,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:35,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:37,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:37,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:38,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:08:39,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:41,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:42,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:43,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:43,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:45,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:47,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:48,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:48,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:49,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:51,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:51,854 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:08:53,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:08:53,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:53,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:08:55,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:57,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:58,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:08:59,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:08:59,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:01,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:03,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:03,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:04,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:05,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:07,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:08,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:09:09,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:10,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:11,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:13,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:13,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:15,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:15,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:17,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:18,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:19,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:21,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:21,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:21,856 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:09:23,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:09:23,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:25,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:26,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:27,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:28,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:29,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:31,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:32,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:33,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:33,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:35,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:37,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:37,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:38,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:09:39,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:41,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:43,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:43,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:43,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:45,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:47,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:48,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:49,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:49,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:51,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:51,861 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:09:53,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:09:53,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:54,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:09:55,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:57,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:09:58,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:09:59,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:00,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:01,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:03,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:03,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:05,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:05,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:07,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:08,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:10:09,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:11,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:11,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:13,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:13,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:15,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:16,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:17,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:18,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:19,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:21,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:21,862 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:10:21,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:23,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:10:23,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:25,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:27,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:27,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:28,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:29,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:31,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:33,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:33,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:33,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:35,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:37,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:38,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:10:38,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:39,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:41,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:43,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:43,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:44,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:45,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:47,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:48,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:49,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:49,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:51,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:51,864 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:10:53,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:10:53,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:55,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:55,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:10:57,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:10:58,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:10:59,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:00,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:01,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:03,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:03,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:05,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:06,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:07,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:08,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:11:09,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:11,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:12,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:13,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:13,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:15,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:17,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:17,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:18,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:19,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:21,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:21,867 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:11:23,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:11:23,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:23,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:25,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:27,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:28,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:28,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:29,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:31,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:33,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:33,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:34,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:35,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:37,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:38,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:11:39,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:39,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:41,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:43,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:43,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:45,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:45,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:47,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:48,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:49,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:50,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:51,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:51,870 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:11:53,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:11:53,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:55,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:56,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:11:57,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:11:58,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:11:59,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:01,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:01,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:03,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:03,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:05,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:07,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:07,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:08,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:12:09,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:11,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:13,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:13,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:13,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:15,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:17,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:18,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:18,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:19,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:21,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:21,872 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:12:23,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:12:23,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:24,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:25,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:27,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:28,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:29,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:29,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:31,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:33,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:33,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:35,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:35,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:37,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:38,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:12:39,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:40,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:41,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:43,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:43,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:45,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:46,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:47,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:48,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:49,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:51,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:51,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:51,875 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:12:53,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:12:53,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:55,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:57,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:12:57,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:12:58,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:12:59,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:01,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:02,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:03,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:03,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:05,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:07,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:08,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:13:08,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:09,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:11,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:13,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:13,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:13,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:15,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:17,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:18,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:19,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:19,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:21,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:21,878 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:13:23,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:13:23,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:25,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:25,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:27,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:28,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:29,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:30,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:31,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:33,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:33,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:35,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:36,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:37,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:38,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:13:39,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:41,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:41,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:43,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:43,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:45,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:47,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:47,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:48,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:49,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:51,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:51,882 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:13:52,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:53,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:13:53,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:55,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:57,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:13:58,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:13:58,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:13:59,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:01,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:03,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:03,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:03,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:05,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:07,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:08,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:14:09,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:09,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:11,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:13,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:13,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:14,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:15,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:17,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:18,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:19,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:20,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:21,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:21,885 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:14:23,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:14:23,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:25,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:25,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:27,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:28,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:29,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:31,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:31,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:33,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:33,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:35,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:36,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:37,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:38,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:14:39,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:41,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:42,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:43,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:43,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:45,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:47,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:48,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:48,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:49,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:51,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:51,888 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:14:53,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:14:53,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:53,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:55,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:57,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:14:58,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:14:59,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:14:59,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:01,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:03,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:03,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:04,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:05,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:07,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:08,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:15:09,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:10,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:11,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:13,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:13,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:15,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:15,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:17,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:18,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:19,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:21,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:21,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:21,891 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:15:23,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:15:23,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:25,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:26,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:27,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:28,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:29,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:31,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:32,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:33,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:33,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:35,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:37,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:37,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:38,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:15:39,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:41,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:43,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:43,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:43,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:45,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:47,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:48,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:48,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:49,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:51,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:51,893 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:15:53,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:15:53,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:54,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:15:55,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:57,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:15:58,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:15:59,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:00,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:01,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:03,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:03,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:05,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:05,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:07,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:08,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:16:09,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:11,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:11,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:13,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:13,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:15,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:16,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:17,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:18,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:19,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:21,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:21,896 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:16:21,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:23,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:16:23,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:25,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:27,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:27,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:28,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:29,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:31,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:33,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:33,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:33,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:35,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:37,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:38,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:16:38,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:39,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:41,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:43,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:43,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:44,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:45,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:47,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:48,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:49,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:49,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:51,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:51,899 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:16:53,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:16:53,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:55,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:16:55,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:57,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:16:58,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:16:59,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:00,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:01,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:03,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:03,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:05,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:06,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:07,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:08,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:17:09,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:11,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:12,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:13,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:13,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:15,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:17,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:17,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:18,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:19,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:21,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:21,905 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:17:23,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:17:23,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:23,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:25,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:27,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:28,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:28,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:29,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:31,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:33,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:33,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:34,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:35,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:37,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:38,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:17:39,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:39,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:41,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:43,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:43,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:45,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:45,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:47,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:48,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:49,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:50,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:51,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:51,905 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:17:53,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:17:53,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:55,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:56,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:17:57,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:17:58,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:17:59,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:01,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:01,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:03,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:03,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:05,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:07,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:07,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:08,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:18:09,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:11,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:12,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:13,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:13,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:15,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:17,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:18,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:18,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:19,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:21,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:21,907 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:18:23,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:18:23,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:24,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:25,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:27,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:28,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:29,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:29,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:31,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:33,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:33,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:35,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:35,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:37,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:38,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:18:39,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:40,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:40,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:40,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:18:41,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:43,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:43,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:45,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:46,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:47,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:48,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:49,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:51,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:51,910 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:18:51,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:53,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:18:53,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:55,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:57,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:18:57,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:18:58,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:18:59,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:01,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:02,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:03,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:03,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:05,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:07,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:08,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:19:08,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:09,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:11,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:13,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:13,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:14,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:15,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:17,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:18,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:19,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:19,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:21,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:21,915 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:19:23,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:19:23,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:25,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:25,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:27,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:28,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:29,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:30,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:31,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:33,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:33,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:35,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:36,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:37,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:38,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:19:39,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:41,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:41,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:43,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:43,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:45,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:47,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:47,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:48,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:49,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:51,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:51,917 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:19:52,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:53,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:19:53,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:55,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:57,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:19:58,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:19:58,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:19:59,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:01,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:03,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:03,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:03,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:05,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:07,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:08,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:20:09,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:09,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:11,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:13,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:13,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:14,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:15,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:17,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:18,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:19,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:20,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:21,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:21,920 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:20:23,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:20:23,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:25,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:26,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:27,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:28,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:29,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:31,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:31,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:33,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:33,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:35,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:37,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:37,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:38,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:20:39,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:41,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:42,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:43,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:43,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:45,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:47,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:48,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:48,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:49,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:51,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:51,922 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:20:53,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:20:53,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:53,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:55,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:57,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:20:58,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:20:59,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:20:59,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:01,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:03,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:03,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:04,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:05,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:07,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:08,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:21:09,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:10,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:11,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:13,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:13,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:15,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:15,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:17,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:18,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:19,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:21,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:21,925 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:21:21,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:23,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:21:23,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:25,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:26,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:27,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:28,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:29,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:31,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:32,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:33,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:33,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:35,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:37,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:38,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:38,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:21:39,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:41,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:43,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:43,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:43,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:45,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:47,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:48,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:49,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:49,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:51,927 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:21:51,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:53,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:21:53,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:54,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:21:55,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:57,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:21:58,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:21:59,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:00,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:01,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:03,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:03,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:05,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:05,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:07,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:08,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:22:09,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:11,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:11,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:13,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:14,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:16,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:16,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:18,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:18,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:20,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:21,930 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:22:21,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:22,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:23,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:22:24,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:26,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:27,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:28,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:28,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:30,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:32,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:33,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:33,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:34,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:36,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:38,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:38,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:22:38,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:40,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:42,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:43,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:44,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:44,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:46,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:48,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:48,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:22:49,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:50,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:51,933 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:22:52,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:53,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:22:54,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:55,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:22:56,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:58,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:22:58,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:00,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:00,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:02,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:03,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:04,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:06,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:06,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:08,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:08,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:23:10,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:12,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:12,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:13,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:14,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:16,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:17,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:18,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:18,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:20,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:21,936 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:23:22,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:23,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:23:23,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:24,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:26,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:28,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:28,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:28,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:30,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:32,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:33,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:34,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:34,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:36,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:38,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:38,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:23:39,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:40,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:42,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:43,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:44,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:45,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:46,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:48,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:48,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:23:50,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:50,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:51,938 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:23:52,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:53,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:23:54,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:56,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:56,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:23:58,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:23:58,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:00,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:01,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:02,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:03,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:04,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:06,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:07,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:08,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:24:08,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:10,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:12,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:12,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:13,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:14,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:16,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:18,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:18,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:18,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:20,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:21,941 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:24:22,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:23,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:24:24,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:24,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:26,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:28,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:28,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:29,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:30,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:32,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:33,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:34,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:35,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:36,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:38,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:24:38,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:40,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:40,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:42,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:43,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:44,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:46,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:46,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:48,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:48,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:24:50,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:51,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:51,946 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:24:52,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:53,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:24:54,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:56,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:57,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:24:58,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:24:58,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:00,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:02,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:02,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:03,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:04,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:06,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:08,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:25:08,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:08,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:10,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:12,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:13,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:13,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:14,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:16,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:18,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:18,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:19,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:20,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:21,946 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:25:22,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:23,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:25:24,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:24,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:26,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:28,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:28,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:30,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:30,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:32,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:33,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:34,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:36,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:36,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:38,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:25:38,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:40,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:41,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:42,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:43,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:44,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:46,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:47,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:48,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:48,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:25:50,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:51,949 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:25:52,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:52,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:53,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:25:54,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:56,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:58,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:25:58,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:25:58,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:00,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:02,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:03,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:03,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:04,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:06,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:08,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:26:08,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:09,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:10,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:12,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:13,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:14,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:14,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:16,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:18,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:18,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:20,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:20,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:21,952 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:26:22,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:23,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:26:24,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:25,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:26,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:28,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:28,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:30,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:31,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:32,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:33,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:34,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:36,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:36,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:38,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:26:38,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:40,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:42,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:42,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:43,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:44,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:46,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:47,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:48,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:48,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:50,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:51,957 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:26:52,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:53,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:26:53,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:26:54,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:56,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:58,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:26:58,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:26:59,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:00,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:02,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:03,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:04,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:04,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:06,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:08,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:27:08,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:10,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:10,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:12,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:13,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:14,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:15,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:16,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:18,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:18,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:20,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:21,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:21,958 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:27:22,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:23,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:27:24,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:26,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:26,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:28,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:28,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:30,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:32,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:32,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:33,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:34,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:36,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:37,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:38,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:27:38,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:40,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:42,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:43,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:43,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:44,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:46,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:48,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:48,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:50,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:51,960 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:27:52,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:53,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:27:54,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:54,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:27:56,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:58,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:27:58,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:27:59,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:00,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:02,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:03,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:04,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:05,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:06,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:08,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:28:08,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:10,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:11,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:12,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:13,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:14,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:16,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:16,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:18,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:18,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:20,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:21,963 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:28:21,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:22,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:23,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:28:24,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:26,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:27,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:28,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:28,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:30,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:32,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:33,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:33,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:34,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:36,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:38,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:28:38,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:38,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:40,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:42,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:43,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:44,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:44,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:46,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:48,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:48,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:49,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:50,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:51,967 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:28:52,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:53,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:28:54,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:55,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:28:56,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:28:58,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:28:58,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:00,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:00,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:02,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:03,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:04,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:06,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:06,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:08,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:29:08,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:10,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:11,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:12,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:13,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:14,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:16,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:17,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:18,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:18,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:20,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:21,968 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:29:22,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:23,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:23,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:29:24,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:26,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:28,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:28,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:28,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:30,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:32,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:33,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:34,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:34,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:36,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:38,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:29:38,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:39,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:40,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:42,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:43,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:44,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:45,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:46,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:48,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:48,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:50,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:50,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:51,971 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:29:52,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:53,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:29:54,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:56,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:29:56,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:29:58,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:29:58,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:00,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:01,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:02,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:03,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:04,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:06,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:07,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:08,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:30:08,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:10,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:12,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:12,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:13,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:14,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:16,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:18,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:18,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:18,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:20,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:21,974 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:30:22,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:23,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:30:23,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:24,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:26,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:28,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:28,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:29,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:30,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:32,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:33,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:34,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:35,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:36,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:38,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:30:38,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:40,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:40,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:42,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:43,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:44,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:46,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:46,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:48,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:48,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:50,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:51,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:51,977 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:30:52,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:53,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:30:54,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:56,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:30:57,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:30:58,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:30:58,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:00,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:02,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:02,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:03,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:04,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:06,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:08,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:31:08,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:08,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:10,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:12,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:13,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:13,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:14,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:16,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:18,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:18,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:19,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:20,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:21,979 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:31:22,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:23,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:31:24,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:24,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:26,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:28,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:28,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:30,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:30,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:32,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:33,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:34,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:35,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:36,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:38,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:31:38,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:40,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:41,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:42,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:43,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:44,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:46,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:47,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:48,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:48,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:50,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:51,982 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:31:52,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:52,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:53,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:31:54,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:56,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:31:58,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:31:58,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:31:58,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:00,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:02,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:03,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:03,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:04,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:06,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:08,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:32:08,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:09,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:10,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:12,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:13,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:14,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:14,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:16,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:18,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:18,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:20,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:20,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:21,988 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:32:22,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:23,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:32:24,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:25,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:26,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:28,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:28,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:30,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:31,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:32,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:33,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:34,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:36,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:36,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:38,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:32:38,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:40,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:42,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:42,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:43,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:44,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:46,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:47,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:48,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:48,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:50,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:51,989 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:32:52,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:53,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:32:53,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:32:54,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:56,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:58,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:32:58,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:32:58,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:00,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:02,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:03,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:04,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:08,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:33:08,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:10,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:10,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:12,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:13,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:14,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:15,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:16,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:18,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:18,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:20,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:20,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:21,991 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:33:22,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:23,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:33:24,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:26,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:26,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:28,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:28,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:30,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:31,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:32,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:33,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:34,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:36,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:37,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:38,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:33:38,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:40,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:42,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:42,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:43,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:44,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:46,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:48,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:48,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:48,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:50,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:51,995 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:33:52,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:53,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:33:53,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:33:54,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:56,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:58,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:33:58,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:33:59,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:00,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:02,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:03,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:04,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:04,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:06,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:08,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:34:08,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:10,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:10,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:12,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:13,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:14,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:16,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:16,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:18,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:18,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:20,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:21,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:22,000 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:34:22,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:23,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:34:24,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:26,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:27,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:28,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:28,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:30,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:32,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:32,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:33,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:34,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:36,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:38,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:34:38,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:38,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:40,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:42,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:43,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:43,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:44,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:46,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:48,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:48,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:49,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:50,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:52,000 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:34:52,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:53,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:34:54,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:34:54,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:56,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:34:58,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:34:58,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:00,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:00,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:02,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:03,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:04,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:05,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:06,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:08,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:35:08,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:10,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:11,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:12,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:13,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:14,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:16,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:16,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:18,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:18,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:20,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:22,004 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:35:22,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:22,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:23,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:35:24,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:26,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:27,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:28,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:28,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:30,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:32,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:33,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:33,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:34,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:36,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:38,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:35:38,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:38,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:40,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:42,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:43,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:44,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:44,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:46,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:48,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:48,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:50,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:50,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:52,007 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:35:52,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:53,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:35:54,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:55,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:35:56,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:35:58,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:35:58,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:00,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:01,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:02,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:03,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:04,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:06,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:06,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:08,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:36:08,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:10,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:12,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:12,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:13,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:14,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:16,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:17,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:18,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:18,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:20,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:22,011 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:36:22,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:23,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:36:23,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:24,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:26,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:28,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:28,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:28,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:30,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:32,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:33,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:34,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:34,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:36,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:38,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:36:38,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:39,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:40,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:42,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:43,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:44,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:45,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:46,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:48,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:48,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:50,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:50,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:52,012 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:36:52,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:53,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:36:54,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:56,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:36:56,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:36:58,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:36:58,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:00,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:02,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:02,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:03,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:04,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:06,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:07,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:08,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:37:08,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:10,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:12,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:13,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:13,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:14,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:16,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:18,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:18,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:18,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:20,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:22,016 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:37:22,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:23,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:37:24,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:24,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:26,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:28,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:28,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:29,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:30,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:32,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:33,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:34,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:35,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:36,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:38,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:37:38,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:40,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:40,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:42,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:43,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:44,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:46,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:46,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:48,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:48,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:50,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:51,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:52,018 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:37:52,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:53,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:37:54,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:56,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:37:57,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:37:58,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:37:58,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:00,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:02,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:02,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:03,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:04,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:06,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:08,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:38:08,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:08,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:10,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:12,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:13,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:14,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:14,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:16,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:18,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:19,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:19,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:21,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:22,021 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:38:23,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:23,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:38:25,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:25,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:27,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:28,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:29,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:30,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:31,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:33,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:33,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:35,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:36,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:37,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:38,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:38:39,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:41,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:41,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:43,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:43,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:45,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:47,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:47,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:48,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:49,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:51,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:52,024 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:38:53,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:53,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:53,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:38:55,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:57,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:38:58,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:38:58,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:38:59,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:01,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:03,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:03,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:03,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:05,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:07,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:08,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:39:09,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:09,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:11,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:13,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:13,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:14,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:15,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:17,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:18,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:19,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:20,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:21,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:22,027 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:39:23,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:23,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:39:25,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:25,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:27,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:28,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:29,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:31,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:31,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:33,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:33,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:35,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:37,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:37,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:38,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:39:39,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:41,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:42,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:43,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:43,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:45,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:47,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:48,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:48,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:49,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:51,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:52,032 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:39:53,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:53,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:39:53,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:39:55,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:57,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:58,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:39:59,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:39:59,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:01,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:03,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:03,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:04,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:05,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:07,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:08,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:40:09,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:10,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:11,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:13,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:13,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:15,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:15,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:17,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:18,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:19,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:20,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:21,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:22,033 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:40:23,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:40:23,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:25,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:26,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:27,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:28,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:29,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:31,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:31,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:33,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:33,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:35,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:36,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:37,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:38,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:40:39,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:41,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:42,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:43,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:43,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:45,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:47,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:47,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:48,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:49,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:51,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:52,035 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:40:52,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:40:53,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:40:53,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:54,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:40:54,710 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 06:40:54,718 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 06:40:54,728 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 06:40:55,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 06:40:55,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:57,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:40:58,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:40:58,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:03,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:03,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:07,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:08,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:41:09,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:13,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:14,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:17,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:41:17,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:41:18,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 06:41:18,049 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 06:41:18,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:41:18,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_13_1528d88fda0ca905ee0d.table.json 2025-09-18 06:41:18,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 06:41:18,648 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/73qdnmyo-media/table/rollouts_13_1528d88fda0ca905ee0d.table.json 2025-09-18 06:41:19,002 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpkuz0m7m7 2025-09-18 06:41:19,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:19,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:19,728 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2Mjc5OTI1Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MjY2MzYzNg==', 'versionIndex': 11}}} 2025-09-18 06:41:19,728 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 06:41:19,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:19,728 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_13_1528d88fda0ca905ee0d.table.json with policy now 2025-09-18 06:41:20,053 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/op5aqgzc-media/table/rollouts_13_1528d88fda0ca905ee0d.table.json 2025-09-18 06:41:21,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:22,039 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:41:23,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:23,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:41:25,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:25,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:27,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:28,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:29,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:31,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:31,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:33,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:33,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:35,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:36,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:37,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:38,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:41:39,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:41,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:42,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:43,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:43,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:45,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:47,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:47,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:48,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:49,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:51,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:52,042 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:41:53,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:53,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:53,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:41:55,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:57,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:41:58,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:41:58,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:41:59,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:01,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:03,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:03,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:04,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:05,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:07,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:08,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:42:09,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:09,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:11,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:13,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:13,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:15,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:15,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:17,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:18,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:19,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:20,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:21,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:22,046 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:42:23,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:23,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:42:25,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:26,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:27,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:28,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:29,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:31,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:32,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:33,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:33,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:35,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:37,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:37,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:38,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:42:39,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:41,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:43,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:43,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:43,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:45,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:47,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:48,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:48,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:49,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:51,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:52,047 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:42:53,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:53,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:42:54,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:42:54,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:54,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:55,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:55,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:55,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 06:42:55,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:57,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:58,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:42:59,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:42:59,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:01,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:03,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:03,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:05,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:05,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:07,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:08,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:43:09,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:11,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:11,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:13,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:13,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:15,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:16,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:17,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:18,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:19,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:21,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:22,050 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:43:22,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:23,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:23,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:43:25,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:27,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:27,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:28,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:29,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:31,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:33,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:33,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:33,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:35,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:37,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:38,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:43:38,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:39,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:41,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:43,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:43,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:44,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:45,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:47,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:48,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:49,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:49,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:51,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:52,053 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:43:53,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:53,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:43:55,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:43:55,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:57,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:43:58,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:43:59,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:00,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:01,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:03,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:03,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:05,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:06,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:07,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:08,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:44:09,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:11,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:11,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:13,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:13,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:15,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:17,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:17,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:18,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:19,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:21,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:22,058 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:44:23,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:23,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:23,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:44:25,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:27,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:28,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:28,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:29,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:31,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:33,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:33,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:34,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:35,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:37,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:38,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:44:39,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:39,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:41,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:43,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:43,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:45,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:45,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:47,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:48,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:49,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:50,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:51,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:52,058 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:44:53,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:53,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:44:55,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:56,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:44:57,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:44:58,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:44:59,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:01,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:01,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:03,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:03,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:05,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:07,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:07,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:08,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:45:09,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:11,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:12,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:13,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:13,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:15,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:17,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:18,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:18,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:19,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:21,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:22,060 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:45:23,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:45:23,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:23,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:25,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:27,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:28,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:29,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:29,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:31,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:33,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:33,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:34,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:35,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:37,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:38,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:45:39,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:40,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:41,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:43,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:43,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:45,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:46,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:47,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:48,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:49,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:51,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:51,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:52,063 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:45:53,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:45:53,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:55,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:57,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:45:57,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:45:58,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:45:59,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:01,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:02,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:03,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:03,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:05,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:07,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:08,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:08,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:46:09,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:11,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:13,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:13,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:13,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:15,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:17,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:18,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:19,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:19,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:21,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:22,068 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:46:23,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:46:23,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:24,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:25,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:27,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:28,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:29,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:30,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:31,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:33,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:33,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:35,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:35,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:37,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:38,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:46:39,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:41,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:41,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:43,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:43,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:45,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:47,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:47,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:48,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:49,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:51,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:52,069 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:46:52,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:53,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:46:53,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:55,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:57,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:46:57,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:46:58,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:46:59,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:01,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:02,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:03,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:03,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:05,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:07,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:07,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:08,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:47:09,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:11,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:13,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:13,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:13,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:15,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:17,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:18,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:18,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:19,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:21,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:22,072 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:47:23,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:47:23,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:24,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:25,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:27,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:28,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:29,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:29,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:31,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:33,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:33,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:35,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:35,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:37,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:38,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:47:39,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:40,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:41,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:43,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:43,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:45,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:46,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:47,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:48,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:49,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:51,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:51,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:52,074 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:47:53,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:47:53,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:55,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:57,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:47:57,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:47:58,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:47:59,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:01,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:03,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:03,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:03,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:05,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:07,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:08,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:48:08,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:09,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:11,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:13,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:13,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:14,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:15,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:17,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:18,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:19,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:19,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:21,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:22,079 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:48:23,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:48:23,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:25,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:25,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:27,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:28,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:29,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:30,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:31,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:33,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:33,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:35,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:36,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:37,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:38,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:48:39,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:41,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:41,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:43,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:43,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:45,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:47,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:47,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:48,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:49,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:51,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:52,081 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:48:53,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:53,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:48:53,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:55,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:57,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:48:58,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:48:58,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:48:59,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:01,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:03,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:03,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:04,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:05,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:07,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:08,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:49:09,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:09,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:11,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:13,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:13,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:15,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:15,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:17,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:18,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:19,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:20,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:21,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:22,084 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:49:23,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:49:23,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:25,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:26,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:27,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:28,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:29,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:31,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:31,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:33,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:33,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:35,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:37,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:37,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:38,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:49:39,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:41,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:42,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:43,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:43,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:45,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:47,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:48,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:48,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:49,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:51,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:52,087 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:49:53,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:49:53,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:53,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:55,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:57,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:49:58,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:49:59,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:49:59,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:01,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:03,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:03,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:04,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:05,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:07,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:08,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:50:09,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:10,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:11,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:13,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:13,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:15,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:16,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:17,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:18,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:19,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:21,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:21,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:22,090 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:50:23,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:50:23,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:25,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:27,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:27,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:28,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:29,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:31,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:32,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:33,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:33,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:35,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:37,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:38,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:38,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:50:39,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:41,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:43,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:43,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:43,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:45,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:47,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:48,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:49,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:49,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:51,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:52,092 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:50:53,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:50:53,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:54,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:50:55,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:57,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:50:58,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:50:59,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:00,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:01,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:03,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:03,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:05,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:05,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:07,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:08,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:51:09,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:11,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:11,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:13,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:13,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:15,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:16,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:17,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:18,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:19,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:21,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:22,095 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:51:22,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:23,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:51:23,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:25,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:27,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:28,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:28,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:29,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:31,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:33,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:33,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:33,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:35,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:37,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:38,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:51:39,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:39,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:41,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:43,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:43,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:44,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:45,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:47,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:48,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:49,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:50,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:51,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:52,098 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:51:53,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:51:53,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:55,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:51:55,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:57,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:51:58,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:51:59,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:01,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:01,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:03,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:03,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:05,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:06,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:07,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:08,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:52:09,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:11,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:12,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:13,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:13,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:15,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:17,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:17,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:18,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:19,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:21,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:22,101 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:52:23,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:52:23,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:23,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:25,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:27,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:28,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:28,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:29,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:31,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:33,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:33,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:34,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:35,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:37,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:38,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:52:39,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:40,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:41,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:43,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:43,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:45,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:45,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:47,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:48,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:49,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:51,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:51,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:52,103 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:52:53,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:52:53,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:55,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:56,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:52:57,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:52:58,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:52:59,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:01,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:02,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:03,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:03,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:05,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:07,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:07,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:08,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:53:09,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:11,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:13,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:13,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:13,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:15,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:17,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:18,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:18,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:19,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:21,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:22,106 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:53:23,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:53:23,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:24,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:25,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:27,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:28,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:29,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:29,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:31,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:33,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:33,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:35,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:35,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:37,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:38,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:53:39,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:40,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:41,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:43,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:43,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:45,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:46,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:47,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:48,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:49,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:51,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:52,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:52,111 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:53:53,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:53:53,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:55,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:57,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:53:57,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:53:58,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:53:59,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:01,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:03,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:03,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:03,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:05,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:07,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:08,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:54:08,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:09,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:11,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:13,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:13,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:14,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:15,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:17,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:18,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:19,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:20,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:22,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:22,112 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:54:23,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:54:24,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:25,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:26,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:28,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:28,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:30,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:30,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:32,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:33,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:34,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:36,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:36,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:38,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:38,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:54:40,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:41,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:42,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:43,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:44,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:46,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:47,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:48,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:48,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:54:50,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:52,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:52,114 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:54:53,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:53,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:54:54,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:56,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:58,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:54:58,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:54:58,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:00,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:02,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:03,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:04,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:04,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:06,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:08,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:08,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:55:09,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:10,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:12,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:13,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:14,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:15,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:16,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:18,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:18,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:20,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:20,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:22,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:22,117 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:55:23,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:55:24,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:26,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:26,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:28,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:28,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:30,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:31,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:32,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:33,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:34,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:36,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:37,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:38,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:38,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:55:40,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:42,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:42,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:43,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:44,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:46,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:48,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:48,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:48,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:50,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:52,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:52,121 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:55:53,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:55:53,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:55:54,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:56,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:58,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:55:58,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:55:59,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:00,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:02,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:03,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:04,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:05,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:06,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:08,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:08,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:56:10,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:10,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:12,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:13,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:14,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:16,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:16,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:18,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:18,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:20,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:21,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:22,123 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:56:22,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:23,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:56:24,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:26,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:27,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:28,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:28,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:30,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:32,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:32,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:33,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:34,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:36,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:38,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:38,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:38,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:56:40,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:42,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:43,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:43,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:44,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:46,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:48,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:48,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:56:49,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:50,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:52,125 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:56:52,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:53,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:56:54,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:54,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:56:56,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:58,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:56:58,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:00,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:00,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:02,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:03,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:04,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:05,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:06,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:08,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:08,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:57:10,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:11,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:12,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:13,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:14,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:16,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:17,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:18,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:18,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:20,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:22,128 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:57:22,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:22,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:23,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:57:24,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:26,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:28,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:28,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:28,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:30,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:32,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:33,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:33,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:34,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:36,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:38,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:38,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:57:39,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:40,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:42,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:43,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:44,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:44,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:46,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:48,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:48,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:57:50,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:50,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:52,133 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:57:52,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:53,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:57:54,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:55,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:57:56,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:58,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:57:58,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:00,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:01,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:02,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:03,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:04,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:06,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:06,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:08,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:08,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:58:10,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:12,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:12,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:13,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:14,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:16,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:17,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:18,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:18,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:20,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:22,134 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:58:22,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:23,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:58:23,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:24,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:26,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:28,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:28,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:29,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:30,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:32,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:33,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:34,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:34,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:36,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:38,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:38,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:58:40,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:40,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:42,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:43,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:44,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:45,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:46,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:48,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:48,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:58:50,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:51,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:52,137 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:58:52,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:53,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:58:54,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:56,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:56,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:58:58,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:58:58,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:00,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:02,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:02,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:03,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:04,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:06,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:07,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:08,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:08,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:59:10,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:12,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:13,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:13,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:14,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:16,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:18,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:18,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:18,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:20,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:22,140 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:59:22,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:23,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:59:24,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:24,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:26,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:28,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:28,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:29,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:30,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:32,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:33,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:34,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:35,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:36,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:38,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:38,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:59:40,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:40,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:42,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:43,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:44,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:46,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:46,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:48,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:48,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 06:59:50,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:52,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:52,171 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 06:59:52,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:53,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 06:59:54,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:56,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:57,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 06:59:58,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 06:59:58,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:00,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:02,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:03,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:03,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:04,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:06,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:08,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:08,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:00:08,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:10,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:12,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:13,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:14,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:14,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:16,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:18,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:18,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:19,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:20,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:22,173 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:00:22,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:23,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:00:24,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:25,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:26,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:28,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:28,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:30,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:30,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:32,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:33,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:34,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:36,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:36,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:38,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:38,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:00:40,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:41,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:42,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:43,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:44,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:46,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:47,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:48,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:48,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:00:50,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:52,176 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:00:52,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:53,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:53,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:00:54,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:56,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:58,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:00:58,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:00:58,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:00,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:02,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:03,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:04,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:04,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:06,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:08,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:01:08,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:09,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:10,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:12,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:13,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:14,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:15,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:16,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:18,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:18,692 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:20,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:20,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:22,179 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:01:22,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:23,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:01:24,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:26,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:26,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:28,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:28,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:30,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:31,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:32,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:33,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:34,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:36,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:37,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:38,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:01:38,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:40,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:42,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:42,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:43,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:44,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:46,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:48,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:48,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:48,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:50,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:52,181 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:01:52,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:53,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:01:53,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:01:54,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:56,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:58,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:01:58,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:01:59,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:00,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:02,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:03,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:04,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:04,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:06,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:08,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:02:08,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:10,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:10,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:12,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:13,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:14,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:16,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:16,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:18,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:18,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:20,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:21,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:22,184 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:02:22,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:23,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:02:24,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:26,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:27,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:28,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:28,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:30,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:32,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:32,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:33,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:34,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:36,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:38,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:38,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:02:38,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:40,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:42,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:43,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:43,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:44,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:46,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:48,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:48,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:02:49,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:50,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:52,187 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:02:52,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:53,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:02:54,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:54,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:02:56,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:58,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:02:58,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:00,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:00,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:02,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:03,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:04,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:05,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:06,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:08,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:03:08,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:10,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:11,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:12,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:13,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:14,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:16,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:16,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:18,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:18,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:20,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:22,192 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:03:22,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:22,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:23,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:03:24,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:26,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:28,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:28,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:28,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:30,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:32,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:33,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:33,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:34,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:36,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:38,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:03:38,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:39,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:40,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:42,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:43,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:44,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:44,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:46,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:48,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:48,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:03:50,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:50,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:52,193 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:03:52,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:53,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:03:54,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:55,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:03:56,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:58,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:03:58,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:00,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:01,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:02,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:03,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:04,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:06,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:06,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:08,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:04:08,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:10,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:12,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:12,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:13,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:14,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:16,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:17,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:18,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:18,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:20,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:22,195 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:04:22,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:23,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:04:23,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:24,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:26,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:28,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:28,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:28,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:30,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:32,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:33,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:34,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:34,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:36,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:38,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:04:38,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:40,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:40,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:42,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:43,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:44,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:45,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:46,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:48,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:48,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:04:50,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:51,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:52,198 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:04:52,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:53,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:04:54,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:56,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:56,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:04:58,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:04:58,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:00,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:02,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:02,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:03,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:04,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:06,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:07,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:08,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:05:08,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:10,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:12,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:13,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:13,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:14,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:16,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:18,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:18,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:18,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:20,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:22,203 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:05:22,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:23,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:05:24,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:24,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:26,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:28,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:28,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:29,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:30,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:32,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:33,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:34,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:35,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:36,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:38,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:05:38,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:40,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:40,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:42,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:43,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:44,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:46,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:46,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:48,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:48,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:50,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:52,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:52,204 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:05:52,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:53,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:05:54,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:56,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:57,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:05:58,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:05:58,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:05:59,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:05:59,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:06:00,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:02,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:03,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:03,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:04,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:06,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:08,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:06:08,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:08,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:10,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:12,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:13,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:14,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:14,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:16,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:18,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:18,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:19,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:20,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:22,207 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:06:22,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:23,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:06:24,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:25,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:26,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:28,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:28,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:30,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:30,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:32,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:33,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:34,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:36,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:36,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:38,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:06:38,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:40,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:42,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:42,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:43,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:44,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:46,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:47,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:48,746 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:48,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:50,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:52,209 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:06:52,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:53,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:53,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:06:54,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:56,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:06:58,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:06:58,748 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:06:58,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:00,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:02,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:03,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:04,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:04,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:06,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:08,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:07:08,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:09,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:10,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:12,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:13,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:14,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:15,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:16,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:18,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:18,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:20,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:20,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:22,214 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:07:22,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:23,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:07:24,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:26,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:26,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:28,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:28,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:30,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:31,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:32,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:33,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:34,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:36,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:37,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:38,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:07:38,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:40,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:42,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:43,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:43,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:44,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:46,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:48,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:48,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:48,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:50,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:52,215 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:07:52,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:53,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:07:54,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:07:54,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:56,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:58,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:07:58,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:07:59,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:00,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:02,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:03,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:04,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:05,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:06,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:08,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:08:08,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:10,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:10,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:12,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:13,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:14,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:16,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:16,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:18,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:18,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:20,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:21,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:22,218 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:08:22,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:23,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:08:24,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:26,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:27,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:28,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:28,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:30,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:32,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:32,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:33,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:34,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:36,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:38,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:38,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:08:38,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:40,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:42,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:43,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:43,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:44,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:46,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:48,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:48,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:49,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:50,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:52,221 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:08:52,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:53,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:08:54,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:55,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:08:56,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:08:58,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:08:58,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:00,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:00,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:02,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:03,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:04,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:06,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:06,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:08,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:09:08,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:10,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:11,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:12,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:13,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:14,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:16,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:17,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:18,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:18,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:20,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:22,223 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:09:22,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:22,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:23,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:09:24,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:26,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:27,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:28,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:28,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:30,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:32,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:32,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:33,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:34,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:36,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:37,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:38,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:09:38,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:40,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:42,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:43,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:43,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:44,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:46,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:48,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:48,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:49,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:50,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:52,227 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:09:52,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:53,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:09:54,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:09:54,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:56,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:09:58,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:09:58,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:00,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:00,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:02,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:03,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:05,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:05,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:07,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:08,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:10:09,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:11,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:11,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:13,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:13,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:15,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:16,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:17,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:18,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:19,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:21,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:22,229 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:10:22,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:23,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:23,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:10:25,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:27,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:27,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:28,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:29,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:31,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:33,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:33,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:33,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:35,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:37,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:38,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:10:38,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:39,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:41,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:43,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:43,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:44,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:45,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:47,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:48,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:49,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:49,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:51,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:52,234 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:10:53,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:53,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:10:55,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:55,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:10:57,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:10:58,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:10:59,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:01,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:01,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:03,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:03,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:05,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:06,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:07,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:08,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:11:09,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:11,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:12,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:13,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:13,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:15,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:17,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:17,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:18,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:19,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:21,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:22,234 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:11:23,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:23,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:23,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:11:25,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:27,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:28,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:28,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:29,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:31,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:33,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:33,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:34,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:35,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:37,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:38,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:11:39,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:39,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:41,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:43,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:43,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:45,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:45,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:47,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:48,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:49,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:50,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:51,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:52,236 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:11:53,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:53,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:11:55,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:56,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:11:57,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:11:58,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:11:59,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:01,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:01,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:03,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:03,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:05,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:07,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:07,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:08,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:12:09,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:11,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:12,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:13,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:13,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:15,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:17,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:18,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:18,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:19,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:21,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:22,239 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:12:23,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:23,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:12:24,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:25,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:27,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:28,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:29,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:29,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:31,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:33,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:33,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:35,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:35,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:37,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:38,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:12:39,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:40,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:41,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:43,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:43,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:45,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:46,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:47,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:48,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:49,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:51,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:51,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:52,244 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:12:53,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:53,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:12:55,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:57,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:12:57,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:12:58,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:12:59,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:01,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:02,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:03,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:03,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:05,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:07,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:08,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:08,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:13:09,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:11,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:13,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:13,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:13,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:15,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:17,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:18,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:19,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:19,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:21,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:22,246 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:13:23,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:23,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:13:24,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:25,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:27,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:28,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:29,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:30,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:31,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:33,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:33,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:35,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:36,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:37,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:38,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:13:39,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:41,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:41,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:43,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:43,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:45,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:47,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:47,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:48,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:49,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:51,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:52,249 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:13:52,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:53,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:53,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:13:55,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:57,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:13:58,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:13:58,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:13:59,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:01,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:03,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:03,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:03,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:05,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:07,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:08,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:14:09,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:09,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:11,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:13,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:13,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:14,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:15,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:17,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:18,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:19,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:20,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:21,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:22,252 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:14:23,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:23,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:14:25,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:25,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:27,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:28,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:29,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:31,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:31,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:33,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:33,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:35,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:36,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:37,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:38,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:14:39,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:41,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:42,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:43,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:43,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:45,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:47,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:48,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:48,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:49,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:51,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:52,256 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:14:53,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:53,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:14:53,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:55,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:57,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:14:58,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:14:59,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:14:59,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:01,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:03,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:03,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:04,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:05,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:07,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:08,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:15:09,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:10,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:11,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:13,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:13,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:15,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:15,730 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:17,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:18,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:19,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:21,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:21,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:22,257 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:15:23,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:23,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:15:25,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:26,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:27,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:28,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:29,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:31,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:32,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:33,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:33,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:35,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:37,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:37,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:38,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:15:39,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:41,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:43,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:43,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:43,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:45,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:47,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:48,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:48,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:49,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:51,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:52,260 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:15:53,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:53,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:15:54,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:15:55,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:57,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:15:58,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:15:59,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:00,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:01,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:03,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:03,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:05,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:05,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:07,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:08,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:16:09,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:11,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:11,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:13,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:13,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:15,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:16,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:17,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:18,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:19,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:21,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:22,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:22,263 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:16:23,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:23,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:16:25,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:27,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:27,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:28,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:29,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:31,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:33,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:33,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:33,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:35,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:37,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:38,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:16:38,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:39,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:41,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:43,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:43,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:44,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:45,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:47,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:48,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:49,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:49,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:51,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:52,265 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:16:53,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:16:53,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:55,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:55,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:16:57,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:16:58,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:16:59,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:01,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:01,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:03,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:03,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:05,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:06,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:07,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:08,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:17:09,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:11,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:12,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:13,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:13,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:15,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:17,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:17,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:18,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:19,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:21,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:22,269 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:17:23,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:23,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:17:23,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:25,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:27,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:28,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:28,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:29,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:31,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:33,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:33,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:34,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:35,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:37,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:38,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:17:39,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:39,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:41,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:43,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:43,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:45,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:45,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:47,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:48,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:49,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:50,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:51,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:52,272 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:17:53,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:17:53,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:55,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:56,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:17:57,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:17:58,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:17:59,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:01,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:01,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:03,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:03,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:05,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:07,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:07,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:08,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:18:09,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:11,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:13,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:13,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:13,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:15,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:17,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:18,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:18,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:19,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:21,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:22,277 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:18:23,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:18:23,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:24,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:25,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:27,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:28,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:29,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:29,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:31,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:33,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:33,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:35,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:35,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:37,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:38,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:18:39,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:40,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:41,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:43,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:43,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:45,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:46,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:47,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:48,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:49,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:51,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:51,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:52,277 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:18:53,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:18:53,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:55,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:57,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:18:57,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:18:58,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:18:59,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:01,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:02,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:03,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:03,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:05,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:07,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:08,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:19:08,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:09,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:11,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:13,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:13,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:13,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:15,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:17,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:18,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:19,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:19,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:21,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:22,280 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:19:23,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:19:23,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:25,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:25,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:27,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:28,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:29,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:30,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:31,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:33,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:33,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:35,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:36,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:37,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:38,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:19:39,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:41,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:41,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:43,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:43,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:45,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:47,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:47,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:48,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:49,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:51,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:52,280 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:19:52,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:53,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:19:53,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:55,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:57,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:19:57,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:19:58,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:19:59,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:01,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:02,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:03,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:03,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:05,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:07,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:08,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:08,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:20:09,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:11,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:13,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:13,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:13,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:15,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:17,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:18,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:18,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:19,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:21,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:22,286 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:20:23,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:20:23,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:23,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:25,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:27,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:28,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:29,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:29,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:31,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:33,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:33,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:34,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:35,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:37,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:38,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:20:39,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:39,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:41,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:43,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:43,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:45,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:45,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:47,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:48,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:49,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:50,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:51,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:52,287 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:20:53,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:20:53,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:55,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:56,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:20:56,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:20:56,676 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 07:20:56,683 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 07:20:56,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 07:20:56,693 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 07:20:57,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:20:58,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:20:58,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:01,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:04,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:06,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:09,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:09,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:21:09,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:11,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:15,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:16,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:18,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:21:18,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:21:18,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 07:21:18,966 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 07:21:18,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:21:19,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:19,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_14_219cd90375b01ac0f3dc.table.json 2025-09-18 07:21:19,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 07:21:19,834 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpoh8pbtla 2025-09-18 07:21:20,061 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/fhk22aed-media/table/rollouts_14_219cd90375b01ac0f3dc.table.json 2025-09-18 07:21:20,826 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2Mjg5NTIyMw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2Mjc5OTI1Mg==', 'versionIndex': 12}}} 2025-09-18 07:21:20,826 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 07:21:20,826 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_14_219cd90375b01ac0f3dc.table.json with policy now 2025-09-18 07:21:21,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:21,197 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/m1frgkyn-media/table/rollouts_14_219cd90375b01ac0f3dc.table.json 2025-09-18 07:21:21,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:22,289 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:21:22,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:23,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:21:23,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:25,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:27,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:27,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:29,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:29,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:31,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:33,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:33,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:34,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:35,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:37,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:38,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:21:38,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:39,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:41,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:43,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:44,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:44,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:45,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:47,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:49,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:49,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:49,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:51,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:52,292 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:21:53,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:21:53,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:55,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:21:55,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:57,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:21:59,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:21:59,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:01,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:01,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:03,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:04,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:05,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:06,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:07,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:08,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:22:09,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:11,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:12,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:13,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:14,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:15,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:17,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:17,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:19,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:19,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:21,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:22,295 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:22:23,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:23,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:22:23,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:25,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:27,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:28,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:29,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:29,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:31,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:33,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:34,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:34,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:35,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:37,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:38,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:22:39,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:39,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:41,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:43,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:44,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:45,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:45,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:47,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:49,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:49,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:50,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:51,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:52,300 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:22:53,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:22:53,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:55,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:56,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:22:57,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:22:59,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:22:59,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:01,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:02,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:03,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:04,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:05,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:07,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:07,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:08,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:23:09,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:11,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:13,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:13,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:14,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:15,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:17,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:18,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:19,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:19,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:21,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:22,301 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:23:23,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:23:23,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:24,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:25,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:27,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:29,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:29,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:29,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:31,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:33,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:34,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:35,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:35,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:37,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:38,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:23:39,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:40,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:41,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:43,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:44,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:45,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:46,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:47,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:49,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:49,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:51,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:51,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:52,303 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:23:53,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:23:53,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:55,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:57,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:23:57,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:23:59,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:23:59,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:01,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:02,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:03,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:04,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:05,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:07,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:08,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:24:08,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:09,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:11,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:13,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:14,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:14,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:15,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:17,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:19,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:19,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:19,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:21,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:22,306 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:24:23,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:24:23,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:25,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:25,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:27,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:29,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:29,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:30,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:31,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:33,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:34,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:35,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:36,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:37,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:38,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:24:39,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:41,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:41,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:43,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:44,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:45,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:47,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:47,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:49,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:49,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:51,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:52,311 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:24:52,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:53,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:24:53,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:55,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:57,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:24:57,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:24:59,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:24:59,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:01,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:02,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:03,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:04,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:05,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:07,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:08,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:08,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:25:09,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:11,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:13,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:13,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:14,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:15,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:17,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:19,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:19,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:19,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:21,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:22,312 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:25:23,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:25:23,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:24,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:25,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:27,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:29,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:29,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:30,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:31,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:33,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:34,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:35,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:35,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:37,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:38,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:25:39,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:41,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:41,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:43,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:44,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:45,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:46,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:48,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:49,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:25:50,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:52,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:52,315 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:25:52,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:53,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:25:54,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:56,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:57,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:25:58,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:25:59,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:00,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:02,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:03,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:04,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:04,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:06,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:08,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:08,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:26:09,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:10,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:12,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:14,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:14,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:14,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:16,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:18,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:19,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:20,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:20,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:22,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:22,318 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:26:23,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:26:24,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:25,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:26,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:28,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:29,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:30,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:31,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:32,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:34,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:34,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:36,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:36,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:38,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:38,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:26:40,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:42,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:42,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:44,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:44,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:46,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:47,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:48,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:49,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:26:50,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:52,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:52,323 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:26:53,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:53,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:26:54,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:56,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:58,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:26:58,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:26:59,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:00,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:02,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:04,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:04,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:04,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:06,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:08,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:08,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:27:09,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:10,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:12,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:14,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:14,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:15,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:16,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:18,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:19,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:20,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:21,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:22,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:22,323 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:27:23,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:27:24,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:26,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:26,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:28,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:29,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:30,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:32,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:32,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:34,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:34,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:36,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:37,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:38,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:38,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:27:40,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:42,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:43,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:44,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:44,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:46,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:48,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:48,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:49,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:50,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:52,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:52,327 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:27:53,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:27:54,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:54,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:27:56,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:58,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:27:59,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:27:59,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:00,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:02,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:04,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:04,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:05,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:06,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:08,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:08,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:28:10,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:10,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:12,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:14,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:14,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:16,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:16,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:18,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:19,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:20,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:21,915 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:22,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:22,330 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:28:23,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:28:24,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:26,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:27,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:28,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:29,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:30,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:32,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:32,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:34,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:34,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:36,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:38,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:38,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:28:38,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:40,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:42,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:44,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:44,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:44,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:46,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:48,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:49,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:28:49,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:50,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:52,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:52,335 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:28:53,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:28:54,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:55,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:28:56,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:58,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:28:59,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:00,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:00,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:02,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:04,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:04,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:06,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:06,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:08,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:08,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:29:10,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:11,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:12,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:14,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:14,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:16,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:17,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:18,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:19,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:20,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:22,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:22,336 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:29:23,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:23,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:29:24,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:26,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:28,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:28,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:29,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:30,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:32,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:33,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:34,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:34,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:36,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:38,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:38,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:29:39,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:40,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:42,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:44,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:44,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:44,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:46,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:48,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:49,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:29:50,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:50,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:52,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:52,340 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:29:53,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:29:54,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:56,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:29:56,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:58,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:29:59,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:00,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:01,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:02,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:04,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:04,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:06,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:07,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:08,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:08,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:30:08,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,746 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:08,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:09,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:30:10,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:12,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:12,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:14,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:14,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:16,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:18,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:18,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:19,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:20,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:22,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:22,342 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:30:23,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:30:23,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:24,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:26,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:28,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:29,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:29,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:30,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:32,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:34,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:34,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:34,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:36,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:38,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:38,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:30:40,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:40,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:42,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:44,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:44,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:46,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:46,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:48,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:49,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:30:50,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:51,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:52,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:52,345 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:30:53,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:30:54,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:56,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:57,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:30:58,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:30:59,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:00,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:02,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:02,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:04,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:04,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:06,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:08,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:08,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:08,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:31:10,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:12,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:13,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:14,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:14,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:16,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:18,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:19,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:19,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:20,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:22,348 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:31:22,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:23,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:31:24,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:24,905 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:26,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:28,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:29,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:30,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:30,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:32,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:34,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:34,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:35,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:36,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:38,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:38,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:31:40,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:41,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:42,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:44,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:44,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:46,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:47,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:48,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:49,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:31:50,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:52,351 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:31:52,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:52,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:53,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:31:54,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:56,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:58,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:31:58,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:31:59,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:00,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:02,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:03,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:04,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:04,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:06,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:08,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:08,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:32:09,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:10,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:12,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:14,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:14,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:14,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:16,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:18,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:19,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:20,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:20,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:22,354 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:32:22,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:23,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:32:24,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:25,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:26,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:28,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:29,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:30,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:31,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:32,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:34,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:34,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:36,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:36,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:38,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:38,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:32:40,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:42,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:42,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:44,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:44,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:46,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:47,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:48,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:49,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:32:50,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:52,357 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:32:52,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:53,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:32:53,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:54,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:56,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:58,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:32:59,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:32:59,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:00,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:02,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:04,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:04,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:04,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:06,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:08,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:33:08,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:10,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:10,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:12,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:14,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:14,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:15,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:16,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:18,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:19,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:20,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:21,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:22,360 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:33:22,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:23,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:33:24,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:26,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:26,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:28,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:29,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:30,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:32,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:32,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:34,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:34,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:36,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:37,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:38,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:33:38,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:40,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:42,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:43,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:44,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:44,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:46,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:48,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:48,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:49,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:50,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:52,363 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:33:52,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:53,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:33:54,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:33:54,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:56,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:58,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:33:59,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:33:59,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:00,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:02,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:04,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:04,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:05,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:06,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:08,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:34:08,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:10,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:11,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:12,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:14,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:14,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:16,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:16,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:18,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:19,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:20,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:22,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:22,368 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:34:22,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:23,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:34:24,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:26,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:27,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:28,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:29,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:30,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:32,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:33,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:34,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:34,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:36,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:38,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:34:38,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:38,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:40,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:42,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:44,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:44,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:44,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:46,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:48,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:49,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:34:49,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:50,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:52,368 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:34:52,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:53,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:34:54,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:55,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:34:56,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:58,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:34:59,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:00,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:00,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:02,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:04,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:04,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:06,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:06,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:08,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:35:08,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:10,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:11,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:12,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:14,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:14,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:16,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:17,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:18,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:19,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:20,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:22,372 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:35:22,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:23,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:23,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:35:24,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:26,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:28,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:28,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:29,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:30,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:32,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:34,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:34,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:34,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:36,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:38,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:35:38,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:39,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:40,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:42,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:44,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:44,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:45,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:46,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:48,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:49,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:35:50,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:50,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:52,374 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:35:52,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:53,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:35:54,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:56,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:35:56,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:58,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:35:59,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:00,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:01,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:02,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:04,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:04,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:06,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:07,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:08,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:36:08,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:10,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:12,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:12,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:14,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:14,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:16,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:18,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:18,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:19,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:20,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:22,379 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:36:22,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:23,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:36:23,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:24,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:26,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:28,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:29,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:29,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:30,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:32,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:34,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:34,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:34,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:36,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:38,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:36:38,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:40,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:40,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:42,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:44,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:44,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:46,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:46,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:48,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:49,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:36:50,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:51,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:52,379 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:36:52,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:53,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:36:54,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:56,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:57,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:36:58,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:36:59,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:00,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:02,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:02,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:04,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:04,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:06,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:08,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:08,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:37:08,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:10,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:12,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:13,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:14,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:14,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:16,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:18,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:19,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:19,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:20,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:22,382 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:37:22,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:23,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:37:24,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:24,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:26,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:28,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:29,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:30,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:30,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:32,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:34,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:34,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:35,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:36,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:38,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:37:38,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:40,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:41,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:42,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:44,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:44,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:46,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:46,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:48,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:49,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:37:50,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:52,385 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:37:52,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:52,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:53,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:37:54,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:56,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:58,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:37:58,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:37:59,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:00,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:02,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:03,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:04,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:04,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:06,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:08,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:38:08,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:09,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:10,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:12,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:14,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:14,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:14,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:16,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:18,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:19,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:20,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:20,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:22,389 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:38:22,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:23,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:38:24,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:25,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:26,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:28,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:29,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:30,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:31,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:32,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:34,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:34,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:36,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:36,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:38,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:38:38,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:40,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:42,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:42,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:44,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:44,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:46,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:47,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:48,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:49,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:38:50,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:52,390 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:38:52,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:53,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:53,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:38:54,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:56,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:58,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:38:58,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:38:59,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:00,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:02,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:04,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:04,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:04,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:06,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:08,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:39:08,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:10,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:10,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:12,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:14,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:14,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:15,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:16,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:18,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:19,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:20,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:21,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:22,393 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:39:22,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:23,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:39:24,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:26,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:26,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:28,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:29,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:30,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:32,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:32,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:34,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:34,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:36,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:37,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:38,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:39:38,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:40,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:42,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:43,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:44,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:44,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:46,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:48,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:48,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:49,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:50,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:52,397 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:39:52,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:53,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:39:54,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:39:54,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:56,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:58,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:39:59,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:39:59,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:00,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:02,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:04,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:04,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:05,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:06,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:08,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:40:08,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:10,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:10,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:12,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:14,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:14,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:16,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:16,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:18,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:19,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:20,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:22,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:22,399 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:40:22,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:23,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:40:24,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:26,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:27,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:28,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:29,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:30,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:32,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:33,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:34,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:34,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:36,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:38,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:40:38,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:38,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:40,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:42,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:44,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:44,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:44,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:46,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:48,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:49,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:40:49,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:50,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:52,402 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:40:52,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:53,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:40:54,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:55,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:40:56,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:58,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:40:59,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:00,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:00,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:02,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:04,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:04,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:06,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:06,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:08,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:41:08,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:10,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:11,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:12,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:14,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:14,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:16,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:17,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:18,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:19,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:20,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:22,405 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:41:23,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:23,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:23,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:41:25,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:27,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:28,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:29,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:29,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:31,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:33,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:34,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:34,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:35,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:37,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:38,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:41:39,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:39,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:41,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:43,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:44,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:45,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:45,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:47,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:49,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:49,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:41:50,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:51,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:52,409 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:41:53,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:53,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:41:55,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:56,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:41:57,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:59,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:41:59,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:01,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:01,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:03,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:04,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:05,046 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:07,048 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:07,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:08,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:42:09,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:11,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:12,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:13,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:14,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:15,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:17,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:18,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:19,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:19,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:21,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:22,411 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:42:23,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:23,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:42:23,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:25,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:27,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:29,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:29,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:29,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:31,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:33,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:34,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:34,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:35,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:37,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:38,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:42:39,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:40,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:41,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:43,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:44,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:45,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:46,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:47,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:49,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:49,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:42:51,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:51,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:52,414 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:42:53,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:53,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:42:55,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:57,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:57,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:42:59,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:42:59,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:01,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:02,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:03,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:04,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:05,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:07,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:08,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:08,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:43:09,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:11,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:13,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:13,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:14,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:15,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:17,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:19,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:19,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:19,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:21,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:22,417 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:43:23,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:23,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:43:24,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:25,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:27,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:29,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:29,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:30,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:31,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:33,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:34,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:35,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:35,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:37,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:38,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:43:39,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:41,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:41,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:43,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:44,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:45,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:47,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:47,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:49,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:49,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:43:51,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:52,444 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:43:52,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:53,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:53,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:43:55,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:57,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:58,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:43:59,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:43:59,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:01,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:03,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:03,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:04,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:05,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:07,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:08,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:44:09,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:09,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:11,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:13,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:14,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:14,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:15,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:17,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:19,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:19,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:20,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:21,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:22,448 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:44:23,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:23,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:44:25,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:25,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:27,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:29,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:29,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:31,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:31,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:33,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:34,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:35,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:36,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:37,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:38,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:44:39,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:41,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:42,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:43,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:44,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:45,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:47,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:47,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:49,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:49,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:44:51,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:52,451 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:44:53,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:53,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:44:53,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:55,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:57,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:59,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:44:59,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:44:59,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:01,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:03,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:04,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:04,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:05,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:07,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:08,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:45:09,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:10,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:11,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:13,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:14,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:15,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:15,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:17,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:19,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:19,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:21,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:21,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:22,454 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:45:23,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:23,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:45:25,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:26,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:27,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:29,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:29,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:31,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:32,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:33,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:34,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:35,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:37,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:37,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:38,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:45:39,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:41,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:43,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:43,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:44,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:45,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:47,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:48,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:49,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:49,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:51,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:52,458 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:45:53,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:53,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:45:54,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:45:55,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:57,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:59,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:45:59,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:45:59,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:01,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:03,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:04,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:05,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:05,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:07,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:08,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:46:09,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:10,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:11,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:13,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:14,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:15,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:16,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:17,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:19,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:19,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:21,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:22,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:22,459 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:46:23,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:23,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:46:25,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:27,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:27,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:29,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:29,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:31,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:33,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:33,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:34,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:35,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:37,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:38,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:46:38,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:39,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:41,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:43,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:44,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:44,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:45,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:47,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:49,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:49,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:46:49,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:51,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:52,462 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:46:53,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:53,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:46:55,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:46:55,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:57,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:59,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:46:59,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:00,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:01,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:03,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:04,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:05,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:06,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:07,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:08,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:47:09,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:11,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:11,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:13,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:14,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:15,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:17,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:17,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:19,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:19,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:21,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:22,465 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:47:22,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:23,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:23,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:47:25,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:27,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:27,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:29,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:29,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:31,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:32,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:33,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:34,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:35,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:37,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:38,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:38,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:47:39,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:41,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:43,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:43,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:44,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:45,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:47,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:49,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:49,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:49,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:47:51,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:52,473 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:47:53,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:53,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:47:54,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:47:55,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:57,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:59,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:47:59,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:00,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:01,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:03,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:04,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:05,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:05,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:07,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:08,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:48:09,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:11,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:11,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:13,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:14,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:15,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:16,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:17,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:19,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:19,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:21,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:22,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:22,473 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:48:23,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:48:23,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:25,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:27,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:27,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:29,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:29,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:31,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:33,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:33,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:34,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:35,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:37,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:38,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:48:39,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:39,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:41,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:43,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:44,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:44,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:45,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:47,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:49,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:49,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:50,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:51,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:52,476 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:48:53,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:48:53,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:55,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:55,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:48:57,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:48:59,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:48:59,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:01,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:01,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:03,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:04,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:05,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:06,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:07,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:08,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:49:09,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:11,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:12,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:13,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:14,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:15,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:17,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:17,829 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:19,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:19,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:21,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:22,479 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:49:23,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:49:23,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:23,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:25,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:27,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:28,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:29,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:29,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:31,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:33,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:34,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:34,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:35,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:37,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:38,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:49:39,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:39,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:41,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:43,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:43,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:45,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:45,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:47,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:48,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:49,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:51,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:51,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:52,482 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:49:53,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:49:53,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:55,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:56,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:49:57,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:49:58,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:49:59,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:01,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:02,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:03,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:03,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:05,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:07,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:07,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:08,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:50:09,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:11,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:13,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:13,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:13,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:15,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:17,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:18,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:18,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:19,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:21,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:22,485 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:50:23,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:50:23,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:24,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:25,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:27,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:28,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:29,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:29,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:31,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:33,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:33,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:35,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:35,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:37,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:38,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:50:39,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:40,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:41,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:43,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:43,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:45,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:46,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:47,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:48,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:49,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:51,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:52,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:52,488 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:50:53,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:50:53,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:55,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:57,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:50:57,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:50:58,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:50:59,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:01,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:03,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:03,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:03,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:05,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:07,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:08,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:51:08,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:09,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:11,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:13,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:13,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:14,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:15,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:17,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:18,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:19,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:19,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:21,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:22,492 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:51:23,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:51:23,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:25,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:25,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:27,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:28,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:29,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:30,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:31,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:33,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:33,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:35,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:36,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:37,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:38,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:51:39,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:41,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:41,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:43,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:43,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:45,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:47,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:47,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:48,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:49,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:51,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:52,494 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:51:52,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:53,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:51:53,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:55,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:57,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:51:58,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:51:58,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:51:59,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:01,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:03,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:03,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:03,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:05,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:07,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:08,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:52:09,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:09,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:11,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:13,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:13,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:15,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:15,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:17,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:18,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:19,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:20,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:21,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:22,497 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:52:23,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:52:23,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:25,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:26,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:27,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:28,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:29,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:31,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:31,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:33,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:33,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:35,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:37,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:37,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:38,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:52:39,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:41,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:42,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:43,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:43,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:45,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:47,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:48,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:48,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:49,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:51,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:52,500 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:52:53,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:52:53,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:53,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:55,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:57,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:52:58,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:52:59,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:52:59,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:01,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:03,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:03,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:04,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:05,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:07,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:08,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:53:09,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:10,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:11,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:13,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:13,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:13,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:13,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:14,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:14,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:14,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 07:53:15,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:16,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:17,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:18,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:19,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:21,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:21,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:22,505 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:53:23,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:53:23,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:25,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:27,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:27,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:28,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:29,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:31,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:32,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:33,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:33,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:35,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:37,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:38,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:38,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:53:39,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:41,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:43,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:43,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:43,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:45,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:47,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:48,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:49,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:49,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:51,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:52,506 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:53:53,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:53:53,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:54,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:53:55,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:57,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:53:58,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:53:59,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:00,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:01,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:03,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:03,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:05,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:05,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:07,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:08,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:54:09,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:11,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:11,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:13,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:13,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:15,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:17,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:17,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:18,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:19,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:21,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:22,509 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:54:22,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:23,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:54:23,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:25,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:27,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:28,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:28,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:29,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:31,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:33,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:33,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:33,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:35,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:37,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:38,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:54:39,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:39,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:41,856 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:43,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:43,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:44,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:45,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:47,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:48,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:49,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:50,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:51,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:52,513 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:54:53,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:54:53,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:55,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:55,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:54:57,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:54:58,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:54:59,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:01,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:01,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:03,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:03,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:05,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:06,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:07,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:08,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:55:09,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:12,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:12,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:13,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:14,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:16,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:18,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:18,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:18,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:19,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:21,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:22,515 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:55:23,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:55:23,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:23,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:25,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:27,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:28,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:29,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:29,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:31,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:33,527 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:33,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:34,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:35,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:37,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:38,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:55:39,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:40,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:41,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:43,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:43,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:45,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:45,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:47,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:48,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:55:49,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:51,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:51,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:52,518 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:55:53,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:55:53,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:55,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:56,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:55:57,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:55:58,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:00,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:02,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:02,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:03,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:04,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:05,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:07,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:07,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:08,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:56:09,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:11,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:13,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:13,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:13,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:15,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:17,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:18,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:18,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:19,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:21,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:22,520 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:56:23,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:56:23,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:24,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:25,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:27,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:28,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:29,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:30,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:32,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:33,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:33,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:35,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:35,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:37,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:38,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:56:39,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:41,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:41,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:43,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:43,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:46,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:46,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:48,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:56:48,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:50,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:52,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:52,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:52,523 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:56:53,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:56:54,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:56,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:57,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:56:58,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:56:58,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:00,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:03,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:03,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:03,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:04,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:06,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:08,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:08,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:57:08,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:10,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:12,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:13,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:14,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:14,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:16,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:18,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:18,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:19,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:20,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:22,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:22,526 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:57:23,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:57:24,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:25,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:26,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:28,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:28,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:30,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:30,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:33,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:33,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:35,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:36,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:37,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:38,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:57:39,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:41,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:42,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:43,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:43,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:44,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:46,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:47,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:48,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:48,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:50,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:52,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:52,528 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:57:53,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:57:53,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:57:54,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:56,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:58,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:57:58,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:57:58,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:00,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:02,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:03,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:04,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:05,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:06,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:08,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:08,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:58:09,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:10,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:12,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:13,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:14,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:15,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:16,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:18,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:18,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:20,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:20,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:22,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:22,531 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:58:23,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:58:24,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:26,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:26,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:28,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:28,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:30,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:31,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:32,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:33,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:34,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:36,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:37,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:38,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:38,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:58:40,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:42,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:42,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:43,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:44,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:46,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:48,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:48,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:48,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:50,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:52,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:52,539 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:58:53,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:58:54,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:58:54,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:56,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:58,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:58:58,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:58:59,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:00,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:02,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:03,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:04,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:05,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:06,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:08,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:08,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:59:10,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:10,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:13,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:13,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:14,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:16,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:16,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:18,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:18,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:20,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:21,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:22,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:22,537 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:59:23,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:59:24,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:26,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:27,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:28,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:28,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:30,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:32,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:32,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:33,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:34,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:36,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:38,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:38,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:38,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:59:40,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:42,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:43,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:43,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:44,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:46,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:48,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:48,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 07:59:49,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:50,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:52,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:52,540 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 07:59:53,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 07:59:54,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:54,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 07:59:56,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:58,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 07:59:58,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:00,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:00,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:02,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:03,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:04,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:06,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:06,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:08,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:08,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:00:10,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:11,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:12,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:13,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:14,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:16,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:16,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:18,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:18,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:20,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:21,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:22,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:22,543 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:00:23,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:00:24,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:26,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:27,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:28,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:28,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:30,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:32,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:32,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:33,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:34,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:36,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:37,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:38,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:38,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:00:40,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:42,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:43,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:43,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:44,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:46,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:48,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:48,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:48,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:50,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:52,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:52,545 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:00:53,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:00:53,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:00:54,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:56,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:57,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:00:57,776 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 08:00:57,784 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 08:00:57,797 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 08:00:58,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 08:00:58,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:00:58,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:00:58,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:03,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:04,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:08,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:01:09,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:10,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:14,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:14,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:14,695 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:19,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:20,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:22,550 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:01:22,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:01:22,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:01:22,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 08:01:22,953 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 08:01:22,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:01:23,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_15_1d2bb76f3e4665f270d8.table.json 2025-09-18 08:01:23,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 08:01:23,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:01:23,862 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpl6_lx80f 2025-09-18 08:01:23,866 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/uq3blckq-media/table/rollouts_15_1d2bb76f3e4665f270d8.table.json 2025-09-18 08:01:24,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:24,565 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2Mjk3ODk1OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2Mjg5NTIyMw==', 'versionIndex': 13}}} 2025-09-18 08:01:24,565 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 08:01:24,566 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_15_1d2bb76f3e4665f270d8.table.json with policy now 2025-09-18 08:01:24,874 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/0fhux6n1-media/table/rollouts_15_1d2bb76f3e4665f270d8.table.json 2025-09-18 08:01:25,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:26,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:28,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:29,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:30,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:30,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:32,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:34,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:34,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:35,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:36,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:38,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:38,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:01:40,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:41,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:42,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:44,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:44,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:46,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:47,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:48,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:49,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:01:50,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:52,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:52,553 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:01:52,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:53,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:01:54,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:56,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:58,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:01:58,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:01:59,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:00,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:02,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:03,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:04,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:04,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:06,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:08,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:08,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:02:09,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:10,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:12,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:14,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:14,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:14,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:16,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:18,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:19,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:20,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:20,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:22,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:22,557 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:02:23,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:02:24,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:25,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:26,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:28,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:29,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:30,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:31,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:32,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:34,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:34,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:36,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:36,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:38,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:38,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:02:40,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:42,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:42,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:44,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:44,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:46,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:47,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:48,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:49,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:02:50,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:52,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:52,559 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:02:53,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:53,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:02:55,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:56,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:58,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:02:59,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:02:59,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:00,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:02,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:04,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:04,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:04,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:06,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:08,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:08,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:03:10,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:10,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:12,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:14,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:14,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:15,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:16,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:18,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:19,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:20,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:21,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:22,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:22,564 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:03:23,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:03:24,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:26,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:26,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:28,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:29,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:30,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:32,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:32,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:34,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:34,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:36,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:37,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:38,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:38,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:03:40,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:42,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:43,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:44,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:44,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:46,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:48,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:48,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:49,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:50,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:52,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:52,565 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:03:53,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:03:54,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:03:54,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:56,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:58,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:03:59,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:03:59,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:00,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:02,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:04,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:04,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:05,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:06,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:08,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:08,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:04:10,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:11,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:12,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:14,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:14,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:16,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:16,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:18,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:19,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:20,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:22,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:22,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:22,569 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:04:23,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:04:24,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:26,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:27,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:28,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:29,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:30,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:32,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:33,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:34,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:34,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:36,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:38,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:38,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:04:38,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:40,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:42,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:44,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:44,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:44,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:46,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:48,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:49,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:04:49,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:50,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:52,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:52,572 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:04:53,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:04:54,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:55,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:04:56,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:58,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:04:59,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:00,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:00,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:02,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:04,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:04,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:06,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:06,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:08,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:08,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:05:10,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:11,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:12,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:14,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:14,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:16,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:17,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:18,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:19,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:20,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:22,576 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:05:22,578 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:22,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:23,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:05:24,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:26,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:28,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:28,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:29,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:30,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:32,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:34,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:34,227 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:34,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:36,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:38,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:38,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:05:39,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:40,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:42,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:44,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:45,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:45,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:46,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:48,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:49,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:05:50,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:50,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:52,578 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:05:52,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:53,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:05:54,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:56,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:05:56,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:58,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:05:59,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:00,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:01,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:02,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:04,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:04,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:06,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:07,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:08,630 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:06:08,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:10,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:12,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:12,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:14,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:14,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:16,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:18,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:18,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:19,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:20,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:22,581 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:06:22,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:23,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:06:23,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:24,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:26,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:28,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:29,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:29,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:30,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:32,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:34,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:34,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:35,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:36,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:38,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:06:38,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:40,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:40,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:42,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:44,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:44,697 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:46,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:46,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:48,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:49,238 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:06:50,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:51,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:52,584 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:06:52,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:53,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:06:54,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:56,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:57,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:06:58,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:06:59,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:00,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:02,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:02,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:04,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:04,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:06,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:08,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:08,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:07:08,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:10,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:12,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:13,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:14,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:14,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:17,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:18,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:19,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:19,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:20,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:22,587 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:07:22,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:23,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:07:24,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:24,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:26,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:28,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:29,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:30,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:30,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:32,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:34,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:34,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:35,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:36,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:38,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:07:38,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:40,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:41,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:42,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:44,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:44,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:46,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:47,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:48,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:49,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:07:50,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:52,589 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:07:52,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:52,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:53,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:07:54,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:56,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:58,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:07:58,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:07:59,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:00,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:02,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:03,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:04,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:04,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:06,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:08,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:08:08,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:09,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:10,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:12,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:14,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:14,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:14,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:16,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:18,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:19,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:20,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:20,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:22,592 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:08:22,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:23,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:08:24,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:25,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:26,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:28,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:29,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:30,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:31,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:32,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:34,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:34,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:36,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:36,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:38,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:08:38,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:40,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:42,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:42,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:44,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:44,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:46,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:48,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:48,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:49,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:50,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:52,595 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:08:52,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:53,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:08:53,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:08:54,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:56,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:58,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:08:59,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:08:59,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:00,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:02,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:04,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:04,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:04,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:06,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:08,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:09:08,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:10,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:10,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:12,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:14,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:14,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:15,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:16,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:18,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:19,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:20,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:21,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:22,598 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:09:22,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:23,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:09:24,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:26,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:26,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:28,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:29,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:30,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:32,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:32,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:34,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:34,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:36,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:37,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:38,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:09:38,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:40,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:42,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:43,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:44,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:44,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:46,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:48,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:48,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:49,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:09:50,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:52,601 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:09:52,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:53,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:09:54,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:09:54,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:56,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:58,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:09:59,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:00,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:00,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:02,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:04,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:04,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:05,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:06,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:08,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:10:08,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:10,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:11,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:12,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:14,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:14,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:16,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:16,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:18,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:19,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:20,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:22,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:22,604 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:10:22,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:23,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:10:24,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:26,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:27,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:28,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:29,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:30,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:32,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:33,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:34,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:34,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:36,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:38,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:10:38,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:38,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:40,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:42,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:44,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:44,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:44,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:46,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:48,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:49,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:10:49,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:50,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:52,607 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:10:52,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:53,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:10:54,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:55,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:10:56,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:59,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:10:59,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:00,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:01,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:03,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:04,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:05,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:06,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:07,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:08,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:11:09,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:11,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:12,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:13,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:14,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:15,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:17,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:17,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:19,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:19,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:21,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:22,609 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:11:22,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:23,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:23,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:11:25,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:27,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:27,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:29,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:29,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:31,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:32,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:33,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:34,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:35,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:37,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:38,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:11:39,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:41,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:43,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:43,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:44,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:45,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:47,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:49,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:49,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:49,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:51,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:52,612 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:11:53,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:11:54,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:54,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:11:56,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:57,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:11:59,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:11:59,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:00,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:01,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:03,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:04,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:05,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:06,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:07,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:08,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:12:09,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:11,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:11,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:13,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:14,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:15,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:17,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:17,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:19,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:19,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:21,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:22,615 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:12:22,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:23,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:23,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:12:25,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:27,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:28,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:29,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:29,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:31,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:33,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:33,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:34,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:35,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:37,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:12:39,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:39,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:41,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:43,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:44,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:44,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:45,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:47,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:49,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:49,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:50,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:51,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:52,619 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:12:53,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:53,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:12:55,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:55,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:12:57,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:12:59,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:12:59,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:01,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:01,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:03,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:04,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:05,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:06,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:07,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:08,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:13:09,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:11,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:12,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:13,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:14,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:15,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:17,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:17,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:19,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:19,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:21,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:22,621 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:13:23,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:23,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:23,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:13:25,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:27,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:29,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:29,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:29,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:31,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:33,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:34,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:34,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:35,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:37,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:13:39,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:40,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:41,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:43,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:44,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:45,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:45,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:47,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:49,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:49,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:51,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:51,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:52,623 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:13:53,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:53,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:13:55,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:56,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:13:57,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:13:59,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:13:59,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:01,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:02,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:03,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:04,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:05,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:07,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:07,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:08,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:14:09,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:11,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:13,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:13,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:14,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:15,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:17,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:18,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:19,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:19,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:21,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:22,627 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:14:23,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:23,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:14:24,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:25,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:27,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:29,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:29,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:30,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:31,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:33,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:34,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:35,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:35,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:37,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:14:39,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:41,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:41,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:43,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:44,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:45,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:46,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:47,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:49,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:49,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:51,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:52,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:52,632 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:14:53,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:53,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:14:55,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:57,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:14:57,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:14:59,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:14:59,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:01,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:03,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:03,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:04,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:05,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:07,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:08,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:15:08,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:09,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:11,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:13,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:14,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:14,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:15,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:17,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:19,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:19,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:19,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:21,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:22,632 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:15:23,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:23,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:15:25,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:25,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:27,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:29,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:29,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:30,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:31,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:33,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:34,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:35,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:36,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:37,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:38,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:15:39,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:41,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:42,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:43,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:44,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:45,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:47,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:47,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:49,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:49,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:51,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:52,635 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:15:52,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:53,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:53,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:15:55,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:57,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:15:57,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:15:59,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:15:59,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:01,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:02,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:03,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:04,323 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:05,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:07,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:08,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:08,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:16:09,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:11,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:13,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:13,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:14,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:15,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:17,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:19,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:19,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:19,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:21,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:22,639 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:16:23,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:23,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:16:24,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:25,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:27,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:29,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:29,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:30,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:31,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:33,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:34,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:35,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:36,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:37,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:38,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:16:39,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:41,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:41,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:43,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:44,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:45,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:47,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:47,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:49,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:49,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:51,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:52,644 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:16:52,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:53,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:53,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:16:55,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:57,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:16:58,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:16:59,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:16:59,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:01,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:03,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:03,760 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:04,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:05,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:07,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:08,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:17:09,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:09,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:11,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:13,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:14,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:14,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:15,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:17,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:19,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:19,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:20,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:21,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:22,644 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:17:23,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:23,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:17:25,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:25,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:26,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:26,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:17:27,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:29,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:29,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:31,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:31,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:33,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:34,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:35,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:37,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:37,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:38,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:17:39,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:41,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:42,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:43,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:44,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:45,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:47,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:48,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:49,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:49,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:51,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:52,647 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:17:53,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:53,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:17:53,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:55,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:57,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:17:59,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:17:59,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:17:59,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:01,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:03,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:04,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:04,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:05,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:07,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:08,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:18:09,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:10,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:11,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:13,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:14,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:15,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:15,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:17,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:19,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:19,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:21,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:21,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:22,650 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:18:23,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:23,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:18:25,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:26,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:27,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:29,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:29,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:31,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:32,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:33,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:34,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:35,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:37,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:38,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:38,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:18:39,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:41,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:43,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:43,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:44,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:45,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:47,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:49,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:49,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:49,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:51,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:52,653 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:18:53,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:53,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:18:54,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:18:55,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:57,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:18:59,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:18:59,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:00,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:01,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:04,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:04,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:05,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:06,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:07,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:08,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:19:09,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:11,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:11,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:13,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:14,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:15,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:16,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:17,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:19,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:19,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:21,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:22,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:22,656 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:19:23,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:23,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:19:25,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:27,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:27,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:29,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:29,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:31,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:33,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:33,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:34,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:35,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:37,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:38,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:19:39,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:39,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:41,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:43,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:44,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:44,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:45,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:47,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:49,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:49,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:50,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:51,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:52,659 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:19:53,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:53,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:19:55,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:55,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:19:57,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:19:59,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:19:59,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:01,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:01,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:04,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:04,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:05,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:06,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:07,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:08,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:20:09,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:11,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:12,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:13,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:14,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:15,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:17,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:17,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:19,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:19,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:21,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:22,661 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:20:23,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:20:23,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:23,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:25,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:27,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:28,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:29,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:29,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:31,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:33,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:34,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:34,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:35,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:37,664 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:38,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:20:39,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:40,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:41,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:43,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:44,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:45,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:45,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:47,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:49,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:49,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:51,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:51,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:52,664 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:20:53,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:20:53,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:55,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:56,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:20:57,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:20:59,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:20:59,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:01,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:02,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:03,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:04,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:06,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:07,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:08,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:21:08,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:09,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:11,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:13,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:13,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:14,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:15,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:17,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:18,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:19,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:19,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:21,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:22,667 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:21:23,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:21:23,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:24,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:25,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:27,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:29,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:29,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:29,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:31,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:33,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:34,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:35,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:36,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:38,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:21:38,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:40,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:40,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:42,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:43,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:44,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:45,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:46,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:47,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:49,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:49,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:51,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:52,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:52,670 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:21:53,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:21:53,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:55,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:57,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:21:57,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:21:59,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:21:59,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:01,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:03,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:03,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:04,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:05,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:07,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:08,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:08,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:22:09,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:11,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:13,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:14,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:14,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:15,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:17,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:19,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:19,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:19,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:21,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:22,675 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:22:23,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:22:23,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:25,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:25,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:27,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:29,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:29,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:30,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:31,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:33,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:34,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:35,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:36,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:37,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:38,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:22:39,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:41,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:41,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:43,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:44,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:45,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:47,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:47,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:49,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:49,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:51,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:52,676 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:22:52,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:53,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:22:53,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:55,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:57,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:22:58,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:22:59,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:22:59,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:01,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:03,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:04,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:04,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:05,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:08,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:23:08,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:09,583 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:09,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:11,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:13,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:14,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:15,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:15,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:17,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:19,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:19,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:20,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:21,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:22,679 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:23:23,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:23:23,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:25,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:26,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:27,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:29,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:29,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:31,753 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:31,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:33,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:34,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:35,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:37,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:37,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:38,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:23:39,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:41,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:42,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:43,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:44,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:45,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:47,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:48,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:49,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:49,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:51,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:52,682 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:23:53,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:23:53,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:53,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:55,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:57,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:23:59,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:23:59,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:23:59,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:01,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:03,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:04,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:04,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:05,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:07,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:08,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:24:09,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:10,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:11,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:13,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:14,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:15,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:16,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:17,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:19,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:19,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:21,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:21,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:22,713 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:24:23,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:24:23,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:25,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:27,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:27,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:29,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:29,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:31,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:32,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:33,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:34,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:35,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:37,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:38,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:38,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:24:40,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:41,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:43,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:43,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:44,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:45,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:47,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:49,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:49,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:49,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:51,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:52,713 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:24:53,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:24:53,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:54,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:24:55,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:57,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:24:59,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:24:59,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:00,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:01,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:03,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:04,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:05,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:05,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:07,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:08,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:25:09,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:11,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:11,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:13,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:14,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:15,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:16,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:17,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:19,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:19,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:21,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:22,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:22,716 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:25:23,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:25:23,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:25,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:27,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:28,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:29,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:30,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:32,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:33,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:34,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:34,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:36,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:38,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:38,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:25:39,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:40,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:43,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:44,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:44,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:44,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:46,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:48,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:49,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:25:50,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:50,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:52,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:52,718 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:25:53,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:25:54,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:55,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:25:56,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:58,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:25:59,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:00,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:01,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:02,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:04,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:04,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:06,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:06,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:08,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:08,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:26:10,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:12,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:12,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:14,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:14,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:16,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:17,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:18,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:19,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:20,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:22,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:22,723 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:26:23,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:26:23,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:24,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:26,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:28,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:28,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:29,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:30,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:32,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:34,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:34,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:34,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:36,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:38,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:38,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:26:40,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:40,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:42,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:44,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:44,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:45,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:46,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:48,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:49,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:26:50,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:51,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:52,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:52,725 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:26:53,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:26:54,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:56,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:56,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:26:58,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:26:59,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:00,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:02,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:02,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:04,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:04,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:06,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:07,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:08,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:08,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:27:10,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:12,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:13,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:14,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:14,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:16,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:18,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:18,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:19,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:20,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:22,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:22,728 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:27:23,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:27:24,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:24,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:26,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:28,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:29,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:29,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:30,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:32,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:34,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:34,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:35,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:36,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:38,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:38,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:27:40,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:40,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:42,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:44,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:44,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:46,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:46,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:48,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:49,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:27:50,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:52,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:52,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:52,731 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:27:53,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:27:54,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:56,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:57,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:27:58,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:27:59,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:00,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:02,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:03,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:04,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:04,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:06,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:08,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:08,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:28:08,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:10,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:12,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:14,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:14,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:14,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:16,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:18,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:19,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:19,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:20,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:22,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:22,733 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:28:23,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:28:24,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:25,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:26,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:28,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:29,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:30,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:30,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:32,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:34,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:34,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:36,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:36,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:38,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:38,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:28:40,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:41,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:42,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:44,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:44,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:46,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:47,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:48,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:49,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:28:50,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:52,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:52,736 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:28:52,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:53,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:28:54,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:56,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:58,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:28:58,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:28:59,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:00,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:02,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:04,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:04,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:04,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:06,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:08,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:08,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:29:09,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:10,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:12,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:14,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:14,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:15,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:16,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:18,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:19,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:20,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:20,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:22,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:22,739 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:29:23,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:29:24,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:26,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:26,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:28,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:29,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:30,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:31,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:32,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:34,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:34,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:36,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:37,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:38,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:38,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:29:40,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:42,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:42,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:44,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:44,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:46,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:48,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:48,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:49,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:50,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:52,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:52,742 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:29:53,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:29:53,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:29:54,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:56,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:58,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:29:59,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:29:59,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:00,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:02,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:04,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:04,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:05,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:06,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:08,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:08,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:30:10,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:10,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:12,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:14,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:14,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:16,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:16,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:18,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:19,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:20,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:21,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:22,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:22,745 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:30:23,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:30:24,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:26,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:27,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:28,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:29,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:30,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:32,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:32,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:34,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:34,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:36,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:38,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:38,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:38,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:30:40,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:42,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:43,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:44,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:44,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:46,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:48,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:49,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:49,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:30:50,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:52,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:52,748 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:30:53,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:30:54,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:54,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:30:56,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:58,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:30:59,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:00,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:00,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:02,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:04,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:04,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:05,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:06,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:08,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:08,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:31:10,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:11,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:12,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:14,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:14,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:16,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:17,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:18,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:19,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:20,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:22,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:22,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:22,750 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:31:23,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:31:24,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:26,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:28,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:28,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:29,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:30,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:32,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:33,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:34,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:34,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:36,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:38,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:38,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:31:39,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:40,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:42,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:44,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:44,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:44,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:46,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:48,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:49,470 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:31:50,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:51,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:52,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:52,755 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:31:53,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:31:54,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:55,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:31:56,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:58,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:31:59,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:00,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:01,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:02,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:04,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:04,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:06,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:06,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:08,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:08,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:32:10,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:12,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:12,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:14,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:14,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:16,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:18,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:18,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:19,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:20,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:22,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:22,757 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:32:23,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:32:23,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:24,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:26,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:28,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:29,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:29,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:30,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:32,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:34,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:34,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:34,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:36,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:38,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:38,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:32:40,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:40,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:42,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:44,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:44,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:45,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:46,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:48,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:49,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:32:50,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:51,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:52,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:52,759 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:32:53,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:32:54,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:56,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:56,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:32:58,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:32:59,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:00,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:02,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:02,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:04,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:04,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:06,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:07,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:08,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:08,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:33:10,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:12,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:13,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:14,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:14,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:16,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:18,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:18,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:19,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:20,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:22,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:22,762 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:33:23,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:33:24,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:24,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:26,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:28,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:29,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:30,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:30,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:32,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:34,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:34,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:35,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:36,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:38,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:38,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:33:40,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:41,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:42,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:44,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:44,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:46,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:46,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:48,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:49,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:33:50,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:52,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:52,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:52,768 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:33:53,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:33:54,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:56,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:57,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:33:58,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:33:59,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:00,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:02,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:03,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:04,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:04,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:06,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:08,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:08,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:34:08,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:10,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:12,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:14,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:14,342 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:14,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:16,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:18,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:19,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:19,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:20,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:22,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:22,769 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:34:23,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:34:25,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:25,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:27,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:28,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:29,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:30,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:30,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:32,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:34,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:34,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:36,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:36,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:38,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:34:38,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:40,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:42,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:42,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:44,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:44,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:46,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:47,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:48,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:49,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:34:50,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:52,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:52,772 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:34:52,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:53,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:34:55,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:56,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:58,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:34:58,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:34:59,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:00,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:02,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:04,232 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:04,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:04,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:06,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:08,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:35:08,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:09,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:10,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:12,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:14,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:14,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:15,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:16,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:18,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:19,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:20,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:20,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:22,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:22,775 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:35:23,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:35:24,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:26,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:27,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:29,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:29,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:30,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:31,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:32,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:34,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:34,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:36,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:37,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:38,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:35:38,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:40,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:42,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:42,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:44,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:44,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:46,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:48,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:48,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:49,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:50,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:52,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:52,778 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:35:53,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:35:54,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:35:54,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:57,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:58,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:35:59,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:35:59,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:00,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:02,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:04,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:04,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:05,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:06,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:08,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:36:08,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:10,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:10,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:12,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:14,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:14,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:16,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:16,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:18,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:19,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:20,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:21,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:22,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:22,780 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:36:23,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:36:24,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:26,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:27,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:28,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:29,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:30,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:32,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:32,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:34,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:34,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:36,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:38,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:38,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:36:38,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:40,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:42,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:43,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:44,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:44,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:46,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:48,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:49,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:49,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:36:50,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:52,784 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:36:52,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:53,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:36:54,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:54,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:36:56,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:58,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:36:59,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:00,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:00,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:02,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:04,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:04,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:06,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:06,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:08,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:37:08,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:10,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:11,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:12,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:14,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:14,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:16,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:17,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:18,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:19,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:20,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:22,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:22,789 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:37:22,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:23,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:37:24,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:26,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:28,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:28,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:29,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:30,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:32,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:33,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:34,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:34,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:36,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:38,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:37:38,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:39,308 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:40,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:42,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:44,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:44,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:44,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:46,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:48,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:49,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:37:50,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:50,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:52,790 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:37:52,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:53,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:37:54,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:55,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:37:56,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:58,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:37:59,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:00,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:01,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:02,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:04,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:04,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:06,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:07,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:08,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:38:08,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:10,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:12,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:12,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:14,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:14,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:16,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:18,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:18,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:19,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:20,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:22,793 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:38:22,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:23,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:38:23,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:24,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:26,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:28,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:29,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:29,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:30,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:32,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:34,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:34,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:34,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:36,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:38,645 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:38:38,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:40,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:40,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:42,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:44,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:44,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:45,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:46,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:48,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:49,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:38:50,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:51,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:52,795 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:38:52,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:53,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:38:54,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:56,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:38:56,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:58,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:38:59,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:00,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:02,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:02,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:04,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:04,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:06,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:07,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:08,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:39:08,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:10,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:12,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:13,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:14,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:14,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:16,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:18,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:18,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:19,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:20,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:22,800 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:39:22,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:23,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:39:24,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:24,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:26,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:28,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:29,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:30,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:30,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:32,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:34,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:34,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:35,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:36,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:38,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:39:38,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:40,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:41,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:42,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:44,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:45,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:46,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:47,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:49,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:49,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:39:51,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:52,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:52,801 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:39:53,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:53,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:39:55,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:57,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:57,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:39:59,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:39:59,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:01,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:03,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:03,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:04,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:05,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:07,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:08,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:40:08,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:09,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:11,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:13,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:14,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:14,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:16,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:17,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:19,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:19,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:19,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:21,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:22,804 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:40:23,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:23,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:40:24,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:25,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:27,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:29,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:29,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:30,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:31,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:33,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:34,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:35,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:35,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:37,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:38,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:40:39,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:40,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:41,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:43,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:44,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:45,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:46,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:47,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:49,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:49,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:51,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:51,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:52,806 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:40:53,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:53,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:40:55,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:56,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:40:57,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:40:59,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:40:59,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:01,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:01,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:01,555 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 08:41:01,562 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 08:41:01,572 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 08:41:02,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 08:41:02,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:03,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:04,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:07,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:09,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:41:12,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:14,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:17,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:17,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:20,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:22,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:22,809 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:41:24,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:41:27,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:28,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:28,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:28,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 08:41:28,985 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 08:41:29,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:29,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_16_84d500de1eb4d485000c.table.json 2025-09-18 08:41:29,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 08:41:29,605 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/n4oswk7e-media/table/rollouts_16_84d500de1eb4d485000c.table.json 2025-09-18 08:41:29,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:29,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:29,867 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpun96mbxw 2025-09-18 08:41:30,906 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MzA2NDk4OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2Mjk3ODk1OQ==', 'versionIndex': 14}}} 2025-09-18 08:41:30,907 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 08:41:30,907 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_16_84d500de1eb4d485000c.table.json with policy now 2025-09-18 08:41:31,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:31,232 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/y68smckn-media/table/rollouts_16_84d500de1eb4d485000c.table.json 2025-09-18 08:41:33,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:33,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:34,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:35,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:37,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:38,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:41:39,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:39,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:40,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:40,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:41,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:41,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 08:41:43,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:44,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:44,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:45,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:47,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:49,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:49,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:41:50,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:51,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:52,814 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:41:53,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:53,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:41:55,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:55,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:41:57,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:59,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:41:59,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:01,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:01,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:03,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:04,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:05,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:07,069 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:07,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:08,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:42:09,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:11,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:12,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:13,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:14,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:15,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:17,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:18,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:19,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:19,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:21,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:22,815 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:42:23,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:23,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:23,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:42:25,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:27,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:29,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:29,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:29,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:31,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:33,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:34,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:34,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:35,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:37,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:38,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:42:39,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:40,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:41,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:43,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:44,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:45,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:45,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:47,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:49,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:49,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:42:51,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:51,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:52,818 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:42:53,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:53,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:42:55,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:56,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:42:57,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:59,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:42:59,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:01,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:02,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:03,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:04,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:05,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:07,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:08,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:08,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:43:10,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:11,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:13,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:13,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:14,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:15,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:17,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:19,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:19,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:19,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:21,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:22,821 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:43:23,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:23,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:43:24,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:25,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:27,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:29,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:29,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:30,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:31,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:33,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:34,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:35,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:35,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:38,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:38,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:43:39,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:41,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:41,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:43,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:44,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:45,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:46,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:47,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:49,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:49,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:43:51,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:52,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:52,826 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:43:53,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:53,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:43:55,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:57,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:57,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:43:59,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:43:59,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:01,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:03,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:03,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:04,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:05,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:07,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:08,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:44:08,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:09,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:11,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:13,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:14,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:14,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:15,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:17,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:19,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:19,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:20,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:21,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:22,827 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:44:23,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:23,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:44:25,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:25,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:27,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:29,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:29,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:31,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:31,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:33,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:34,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:35,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:36,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:37,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:38,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:44:39,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:41,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:42,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:43,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:44,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:45,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:47,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:47,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:49,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:49,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:44:51,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:52,830 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:44:52,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:53,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:53,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:44:55,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:57,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:58,804 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:44:59,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:44:59,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:01,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:03,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:04,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:04,832 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:05,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:07,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:08,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:45:09,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:09,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:11,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:13,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:14,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:15,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:15,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:17,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:19,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:19,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:20,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:21,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:22,835 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:45:23,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:23,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:45:25,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:26,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:27,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:29,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:29,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:31,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:32,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:33,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:34,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:35,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:37,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:37,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:38,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:45:39,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:41,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:43,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:43,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:44,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:45,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:47,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:48,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:49,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:49,840 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:51,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:52,837 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:45:53,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:53,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:45:54,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:45:55,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:57,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:59,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:45:59,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:45:59,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:01,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:03,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:04,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:05,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:05,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:07,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:08,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:46:09,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:10,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:11,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:13,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:14,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:15,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:16,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:17,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:19,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:19,845 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:21,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:21,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:22,840 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:46:23,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:23,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:46:25,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:27,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:27,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:29,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:29,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:31,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:32,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:33,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:34,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:35,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:37,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:38,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:38,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:46:39,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:41,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:43,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:44,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:44,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:45,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:47,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:49,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:49,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:49,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:51,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:52,843 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:46:53,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:53,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:46:55,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:46:55,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:57,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:46:59,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:46:59,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:00,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:01,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:03,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:04,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:05,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:06,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:07,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:08,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:47:09,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:11,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:11,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:13,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:14,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:15,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:17,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:17,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:19,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:19,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:21,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:22,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:22,845 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:47:23,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:23,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:47:25,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:27,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:28,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:29,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:29,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:31,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:33,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:33,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:34,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:35,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:37,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:38,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:47:39,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:39,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:41,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:43,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:44,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:44,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:45,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:47,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:49,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:49,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:50,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:51,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:52,850 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:47:53,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:53,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:47:55,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:56,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:47:57,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:47:59,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:47:59,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:01,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:01,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:03,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:04,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:05,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:07,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:07,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:08,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:48:09,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:11,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:12,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:13,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:14,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:15,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:17,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:18,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:19,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:19,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:21,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:22,852 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:48:23,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:23,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:23,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:48:25,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:27,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:29,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:29,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:29,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:31,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:33,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:34,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:34,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:35,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:37,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:38,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:48:39,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:40,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:41,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:43,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:44,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:45,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:45,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:47,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:49,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:49,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:51,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:51,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:52,855 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:48:53,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:53,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:48:55,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:56,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:48:57,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:48:58,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:48:59,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:01,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:02,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:03,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:03,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:05,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:07,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:08,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:08,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:49:09,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:11,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:13,595 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:13,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:13,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:15,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:17,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:18,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:19,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:19,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:21,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:22,860 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:49:23,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:23,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:49:24,688 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:25,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:27,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:28,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:29,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:30,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:31,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:33,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:33,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:35,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:35,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:37,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:38,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:49:39,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:41,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:41,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:43,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:43,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:45,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:46,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:47,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:48,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:49,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:51,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:52,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:52,862 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:49:53,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:53,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:49:55,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:57,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:49:57,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:49:58,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:49:59,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:01,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:03,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:03,736 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:03,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:05,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:07,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:08,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:50:09,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:09,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:11,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:13,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:13,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:14,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:15,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:17,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:18,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:19,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:20,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:21,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:22,864 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:50:23,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:23,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:50:25,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:25,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:27,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:28,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:29,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:31,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:31,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:33,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:33,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:35,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:36,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:37,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:38,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:50:39,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:41,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:42,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:43,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:43,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:45,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:47,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:47,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:48,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:49,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:51,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:52,867 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:50:52,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:53,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:53,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:50:55,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:57,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:50:57,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:50:58,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:50:59,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:01,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:03,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:03,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:05,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:07,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:08,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:08,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:51:09,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:11,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:13,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:13,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:14,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:15,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:17,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:18,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:19,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:19,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:21,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:22,872 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:51:23,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:23,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:51:25,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:25,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:27,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:28,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:29,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:30,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:31,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:33,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:33,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:35,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:36,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:37,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:38,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:51:39,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:41,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:41,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:43,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:43,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:45,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:47,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:47,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:48,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:49,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:51,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:52,873 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:51:52,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:53,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:53,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:51:55,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:57,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:51:58,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:51:58,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:51:59,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:01,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:03,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:03,902 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:03,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:05,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:07,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:08,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:52:09,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:09,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:11,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:13,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:13,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:15,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:15,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:17,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:18,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:19,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:20,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:21,898 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:22,875 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:52:23,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:52:23,900 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:25,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:26,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:27,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:28,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:29,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:31,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:31,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:33,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:33,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:35,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:37,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:37,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:38,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:52:39,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:41,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:42,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:43,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:43,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:45,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:47,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:48,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:48,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:49,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:51,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:52,879 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:52:53,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:52:53,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:53,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:55,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:57,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:52:58,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:52:59,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:52:59,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:01,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:03,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:03,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:04,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:05,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:07,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:08,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:53:09,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:10,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:11,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:13,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:13,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:15,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:15,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:17,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:18,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:19,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:21,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:21,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:22,885 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:53:23,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:53:23,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:25,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:26,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:27,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:28,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:29,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:31,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:32,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:33,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:33,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:35,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:37,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:38,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:38,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:53:39,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:41,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:43,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:43,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:43,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:45,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:47,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:48,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:53:49,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:49,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:52,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:52,884 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:53:53,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:53:54,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:54,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:53:56,005 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:58,007 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:53:58,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:00,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:00,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:02,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:03,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:04,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:05,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:06,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:08,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:08,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:54:10,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:11,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:12,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:13,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:14,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:16,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:16,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:18,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:18,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:20,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:22,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:22,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:22,888 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:54:23,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:54:24,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:26,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:27,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:28,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:28,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:30,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:32,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:33,501 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:33,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:34,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:36,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:38,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:38,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:54:39,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:40,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:42,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:43,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:44,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:44,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:46,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:48,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:48,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:54:50,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:50,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:52,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:52,889 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:54:53,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:54:54,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:55,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:54:56,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:58,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:54:58,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:00,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:01,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:02,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:03,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:04,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:06,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:06,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:08,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:08,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:55:10,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:12,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:12,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:13,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:14,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:16,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:17,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:18,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:18,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:20,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:22,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:22,895 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:55:22,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:23,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:55:24,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:26,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:28,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:28,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:28,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:30,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:32,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:33,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:34,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:34,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:36,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:38,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:38,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:55:39,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:40,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:42,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:43,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:44,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:45,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:46,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:48,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:48,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:55:50,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:51,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:52,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:52,896 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:55:53,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:55:54,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:56,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:56,612 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:55:58,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:55:58,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:00,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:02,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:02,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:03,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:04,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:06,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:07,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:08,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:08,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:56:10,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:12,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:13,266 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:13,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:14,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:16,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:18,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:18,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:18,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:20,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:22,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:22,898 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:56:23,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:56:24,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:24,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:26,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:28,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:28,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:29,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:30,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:32,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:33,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:34,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:35,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:36,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:38,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:38,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:56:40,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:41,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:42,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:43,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:44,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:46,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:46,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:48,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:48,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:56:50,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:52,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:52,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:52,902 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:56:53,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:56:54,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:56,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:57,617 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:56:58,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:56:58,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:00,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:02,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:03,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:03,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:04,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:06,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:08,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:08,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:08,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:57:10,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:12,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:13,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:14,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:14,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:16,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:18,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:18,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:19,783 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:20,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:22,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:22,905 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:57:23,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:57:24,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:25,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:26,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:28,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:28,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:30,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:30,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:32,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:33,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:34,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:36,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:36,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:38,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:38,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:57:40,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:41,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:42,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:43,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:44,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:46,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:47,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:48,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:48,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:57:50,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:52,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:52,908 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:57:52,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:53,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:57:54,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:56,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:58,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:57:58,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:57:58,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:00,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:02,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:03,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:04,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:04,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:06,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:08,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:08,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:58:09,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:10,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:12,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:13,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:14,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:15,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:16,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:18,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:18,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:20,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:20,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:22,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:22,910 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:58:23,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:58:24,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:26,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:26,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:28,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:28,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:30,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:31,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:32,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:33,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:34,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:36,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:37,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:38,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:38,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:58:40,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:42,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:42,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:43,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:44,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:46,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:48,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:48,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:48,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:50,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:52,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:52,914 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:58:53,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:58:53,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:58:55,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:56,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:58,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:58:58,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:58:59,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:00,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:02,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:03,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:04,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:04,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:06,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:08,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:08,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:59:10,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:10,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:12,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:13,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:14,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:16,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:16,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:18,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:18,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:20,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:21,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:22,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:22,916 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:59:23,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:59:24,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:26,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:27,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:28,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:28,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:30,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:32,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:32,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:33,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:34,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:36,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:38,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:38,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:38,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:59:40,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:42,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:43,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:43,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:44,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:46,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:48,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:48,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 08:59:49,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:50,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:52,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:52,920 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 08:59:53,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 08:59:54,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 08:59:55,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:56,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:58,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 08:59:58,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:00,369 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:00,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:02,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:03,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:04,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:05,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:06,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:08,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:08,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:00:10,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:11,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:12,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:13,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:14,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:16,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:16,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:18,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:18,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:20,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:22,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:22,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:22,923 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:00:23,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:00:24,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:26,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:28,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:28,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:28,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:30,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:32,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:33,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:33,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:34,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:36,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:38,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:38,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:00:39,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:40,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:42,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:43,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:44,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:44,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:46,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:48,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:48,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:50,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:50,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:52,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:52,928 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:00:53,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:00:55,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:55,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:00:57,491 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:00:58,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:00:59,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:01,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:01,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:03,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:03,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:04,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:06,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:06,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:08,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:08,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:01:10,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:12,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:12,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:14,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:14,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:16,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:17,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:18,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:19,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:20,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:22,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:22,928 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:01:22,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:23,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:01:24,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:27,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:28,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:28,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:29,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:30,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:32,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:34,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:34,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:34,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:36,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:38,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:38,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:01:40,001 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:40,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:42,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:44,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:44,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:45,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:46,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:48,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:49,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:01:50,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:51,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:52,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:52,932 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:01:53,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:01:54,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:56,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:56,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:01:58,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:01:59,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:00,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:02,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:02,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:04,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:04,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:06,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:07,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:08,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:08,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:02:10,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:12,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:13,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:14,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:14,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:16,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:18,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:18,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:19,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:20,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:22,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:22,934 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:02:23,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:02:24,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:24,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:26,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:28,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:29,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:29,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:30,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:32,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:34,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:34,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:35,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:36,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:38,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:38,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:02:40,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:40,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:42,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:44,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:44,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:46,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:46,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:48,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:49,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:02:50,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:52,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:52,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:52,939 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:02:53,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:02:54,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:56,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:57,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:02:58,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:02:59,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:00,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:02,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:03,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:04,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:04,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:06,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:08,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:08,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:08,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:03:10,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:12,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:14,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:14,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:14,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:16,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:18,662 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:19,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:19,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:20,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:22,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:22,941 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:03:23,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:03:24,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:25,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:26,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:28,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:29,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:30,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:31,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:33,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:34,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:34,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:36,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:36,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:38,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:38,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:03:40,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:41,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:42,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:44,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:44,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:46,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:47,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:48,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:49,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:03:50,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:52,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:52,943 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:03:52,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:53,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:03:54,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:56,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:58,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:03:58,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:03:59,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:01,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:02,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:04,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:04,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:04,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:06,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:08,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:08,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:04:09,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:10,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:12,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:14,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:14,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:15,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:16,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:18,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:19,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:20,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:20,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:22,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:22,946 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:04:23,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:04:24,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:26,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:26,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:28,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:29,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:30,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:31,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:32,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:34,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:34,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:36,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:37,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:38,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:38,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:04:40,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:42,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:42,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:44,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:44,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:45,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,770 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,779 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,933 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,971 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:45,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,006 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,061 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:04:46,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:48,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:48,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:49,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:50,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:52,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:52,951 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:04:53,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:04:53,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:04:54,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:56,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:58,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:04:59,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:04:59,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:00,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:02,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:04,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:04,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:05,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:06,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:08,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:08,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:05:10,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:10,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:12,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:14,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:14,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:16,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:16,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:18,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:19,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:20,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:21,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:22,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:22,952 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:05:23,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:05:24,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:26,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:27,234 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:28,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:29,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:30,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:32,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:32,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:34,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:34,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:36,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:38,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:38,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:38,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:05:40,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:42,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:43,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:44,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:44,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:46,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:48,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:49,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:05:49,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:50,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:52,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:52,955 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:05:53,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:05:54,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:54,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:05:56,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:58,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:05:59,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:00,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:00,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:02,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:04,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:04,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:06,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:06,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:08,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:08,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:06:10,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:11,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:12,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:14,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:14,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:16,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:17,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:18,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:19,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:20,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:22,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:22,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:22,957 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:06:23,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:06:24,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:26,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:28,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:28,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:29,054 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:30,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:32,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:33,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:34,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:34,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:36,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:38,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:06:38,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:39,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:40,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:42,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:44,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:44,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:44,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:46,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:48,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:49,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:06:50,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:50,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:52,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:52,960 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:06:53,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:06:54,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:55,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:06:56,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:58,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:06:59,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:00,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:01,415 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:02,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:04,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:04,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:06,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:06,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:08,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:07:08,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:10,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:12,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:12,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:14,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:14,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:16,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:18,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:18,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:19,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:20,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:22,963 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:07:22,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:23,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:07:23,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:24,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:26,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:28,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:29,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:29,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:30,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:32,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:34,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:34,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:34,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:36,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:38,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:07:38,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:40,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:40,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:42,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:44,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:44,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:45,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:46,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:48,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:49,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:07:50,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:51,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:52,966 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:07:53,000 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:53,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:07:55,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:56,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:07:57,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:59,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:07:59,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:01,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:02,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:03,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:04,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:05,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:07,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:07,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:08,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:08:09,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:11,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:13,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:13,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:14,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:15,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:17,027 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:19,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:19,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:19,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:21,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:22,971 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:08:23,034 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:23,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:08:24,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:25,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:27,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:29,041 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:29,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:30,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:31,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:33,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:34,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:35,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:35,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:37,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:38,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:08:39,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:41,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:41,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:43,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:44,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:45,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:46,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:47,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:49,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:49,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:08:51,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:52,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:52,971 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:08:53,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:53,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:08:55,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:57,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:57,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:08:59,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:08:59,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:01,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:03,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:03,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:04,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:05,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:07,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:08,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:09:08,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:09,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:11,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:13,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:14,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:14,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:15,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:17,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:19,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:19,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:19,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:21,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:22,974 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:09:23,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:23,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:09:25,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:25,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:27,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:29,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:29,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:31,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:31,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:33,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:34,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:35,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:36,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:37,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:38,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:09:40,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:41,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:42,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:43,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:44,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:45,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:47,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:47,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:49,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:49,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:51,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:52,977 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:09:52,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:53,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:53,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:09:55,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:57,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:09:58,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:09:59,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:09:59,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:01,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:03,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:04,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:04,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:05,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:07,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:08,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:10:09,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:09,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:11,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:13,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:14,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:15,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:15,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:17,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:19,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:19,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:20,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:21,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:22,982 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:10:23,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:23,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:10:25,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:26,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:27,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:29,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:29,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:31,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:31,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:33,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:34,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:35,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:37,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:37,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:38,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:10:39,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:42,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:43,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:44,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:44,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:45,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:47,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:48,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:49,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:49,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:51,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:52,982 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:10:53,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:53,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:10:54,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:10:55,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:57,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:59,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:10:59,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:10:59,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:01,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:03,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:04,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:05,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:05,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:07,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:08,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:11:09,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:10,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:11,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:13,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:14,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:15,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:16,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:17,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:19,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:19,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:21,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:21,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:22,985 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:11:23,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:23,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:11:25,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:27,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:27,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:29,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:29,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:31,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:32,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:33,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:34,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:35,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:37,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:38,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:38,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:11:39,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:41,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:43,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:43,993 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:44,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:45,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:47,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:49,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:49,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:49,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:51,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:52,989 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:11:53,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:53,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:11:55,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:11:55,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:57,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:11:59,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:11:59,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:00,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:01,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:03,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:04,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:05,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:06,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:07,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:08,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:12:09,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:11,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:11,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:14,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:14,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:16,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:17,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:17,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:19,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:19,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:21,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:22,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:22,993 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:12:23,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:23,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:12:25,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:27,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:28,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:29,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:29,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:31,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:33,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:33,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:34,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:35,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:37,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:38,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:12:39,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:39,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:41,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:44,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:44,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:44,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:45,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:47,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:49,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:49,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:50,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:51,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:52,994 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:12:53,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:53,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:12:55,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:55,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:12:57,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:12:59,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:12:59,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:01,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:01,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:03,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:04,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:05,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:07,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:07,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:08,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:13:09,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:11,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:12,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:13,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:14,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:15,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:17,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:18,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:19,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:19,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:21,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:22,997 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:13:23,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:23,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:13:23,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:25,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:27,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:29,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:29,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:29,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:31,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:33,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:34,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:34,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:35,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:37,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:38,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:13:39,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:40,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:41,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:43,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:44,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:45,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:46,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:48,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:49,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:49,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:51,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:51,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:52,999 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:13:53,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:53,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:13:55,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:56,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:13:57,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:13:59,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:13:59,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:01,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:02,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:03,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:04,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:05,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:07,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:07,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:08,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:14:09,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:11,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:13,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:13,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:14,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:15,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:17,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:19,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:19,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:19,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:21,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:23,002 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:14:23,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:23,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:14:24,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:25,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:27,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:29,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:29,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:30,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:31,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:33,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:34,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:35,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:35,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:37,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:38,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:14:39,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:41,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:41,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:43,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:44,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:46,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:46,733 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:48,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:49,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:49,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:51,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:52,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:53,004 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:14:53,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:53,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:14:55,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:57,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:14:57,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:14:59,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:14:59,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:01,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:03,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:03,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:04,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:05,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:07,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:08,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:15:08,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:09,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:11,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:13,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:14,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:14,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:15,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:17,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:19,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:19,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:19,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:21,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:23,007 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:15:23,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:23,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:15:25,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:25,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:27,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:29,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:29,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:31,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:31,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:33,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:34,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:35,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:36,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:37,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:38,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:15:39,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:41,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:42,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:43,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:44,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:45,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:47,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:47,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:49,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:49,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:51,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:53,012 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:15:53,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:53,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:53,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:15:55,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:57,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:15:58,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:15:59,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:15:59,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:01,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:03,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:04,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:04,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:05,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:07,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:08,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:16:09,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:09,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:11,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:13,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:14,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:15,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:15,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:18,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:19,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:20,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:20,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:21,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:23,013 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:16:23,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:23,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:16:25,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:26,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:27,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:29,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:29,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:31,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:31,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:33,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:34,160 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:35,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:37,474 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:37,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:38,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:16:39,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:41,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:43,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:43,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:44,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:45,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:47,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:48,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:49,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:49,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:51,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:53,016 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:16:53,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:53,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:16:54,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:55,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:57,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:16:59,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:16:59,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:16:59,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:01,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:03,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:04,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:05,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:05,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:07,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:08,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:17:09,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:10,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:11,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:13,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:14,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:15,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:16,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:17,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:19,168 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:19,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:21,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:21,759 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:23,020 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:17:23,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:23,880 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:17:25,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:27,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:27,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:29,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:29,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:31,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:32,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:33,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:34,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:35,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:37,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:38,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:38,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:17:39,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:41,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:43,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:43,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:44,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:45,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:47,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:49,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:49,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:50,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:51,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:53,060 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:17:53,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:53,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:17:55,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:17:55,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:57,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:17:59,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:17:59,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:00,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:01,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:03,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:04,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:05,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:06,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:07,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:08,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:18:09,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:11,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:11,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:13,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:14,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:15,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:17,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:17,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:19,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:19,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:21,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:22,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:23,061 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:18:23,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:23,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:18:25,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:27,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:28,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:29,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:29,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:31,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:33,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:33,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:34,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:35,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:37,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:38,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:18:39,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:39,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:41,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:43,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:44,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:44,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:45,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:48,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:49,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:49,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:50,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:52,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:53,064 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:18:53,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:53,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:18:55,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:55,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:18:57,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:18:59,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:18:59,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:01,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:01,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:03,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:04,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:05,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:06,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:07,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:08,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:19:09,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:11,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:12,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:13,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:14,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:15,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:17,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:18,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:19,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:19,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:21,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:23,067 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:19:23,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:23,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:23,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:19:25,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:27,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:28,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:29,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:29,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:31,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:33,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:33,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:34,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:35,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:37,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:38,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:38,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:19:39,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:41,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:43,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:44,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:44,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:45,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:47,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:49,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:49,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:49,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:51,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:53,070 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:19:53,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:53,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:19:55,492 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:19:55,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:57,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:19:59,197 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:19:59,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:01,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:01,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:03,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:04,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:05,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:06,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:07,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:08,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:20:09,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:11,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:12,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:13,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:14,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:15,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:17,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:17,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:19,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:19,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:21,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:23,073 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:20:23,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:23,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:20:24,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:25,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:27,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:28,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:29,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:29,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:31,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:33,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:33,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:34,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:35,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:37,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:38,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:38,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:20:39,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:41,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:43,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:44,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:44,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:45,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:47,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:49,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:49,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:49,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:51,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:53,076 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:20:53,883 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:20:53,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:54,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:20:55,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:57,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:20:59,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:20:59,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:00,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:01,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:03,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:04,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:05,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:05,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:07,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:08,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:21:08,100 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 09:21:08,107 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 09:21:08,117 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 09:21:08,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 09:21:09,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:09,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:10,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:21:11,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:16,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:16,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:21,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:21,824 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:21,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:23,078 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:21:25,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:21:27,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:30,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:32,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:21:33,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:21:33,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:21:34,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 09:21:34,244 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 09:21:34,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:21:34,970 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_17_97878f334b2dd769a034.table.json 2025-09-18 09:21:34,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 09:21:35,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:21:36,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:39,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:21:44,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:50,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:21:54,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:21:59,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:05,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:09,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:22:14,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:20,486 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:24,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:22:29,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:35,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:39,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:22:44,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:50,491 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:22:54,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:22:59,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:05,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:09,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:23:14,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:20,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:24,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:23:29,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:35,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:39,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:23:44,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:45,251 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpr2hya2im 2025-09-18 09:23:47,303 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/2hqamb9c-media/table/rollouts_17_97878f334b2dd769a034.table.json 2025-09-18 09:23:50,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:23:54,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:23:59,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:05,502 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:09,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:24:14,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:20,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:24,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:24:29,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:35,506 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:39,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:24:44,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:50,508 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:24:54,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:24:59,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:05,510 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:09,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:25:14,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:20,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:24,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:25:29,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:35,515 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:39,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:25:44,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:50,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:25:54,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:25:56,773 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MzE1MzA5NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MzA2NDk4OA==', 'versionIndex': 15}}} 2025-09-18 09:25:56,774 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 09:25:56,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:25:56,774 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_17_97878f334b2dd769a034.table.json with policy now 2025-09-18 09:25:56,775 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,776 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,777 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,779 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,780 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,781 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,782 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,783 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:56,783 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:25:57,132 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/m3s6ftdd-media/table/rollouts_17_97878f334b2dd769a034.table.json 2025-09-18 09:25:58,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:25:59,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:00,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:01,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:02,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:04,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:05,519 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:06,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:07,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:08,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:09,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:26:10,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:12,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:12,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:14,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:14,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:16,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:18,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:18,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:20,296 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:20,522 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:22,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:23,109 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:26:24,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:24,301 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:24,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:26:26,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:28,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:29,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:29,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:30,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:32,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:34,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:35,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:35,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:36,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:38,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:39,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:26:40,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:40,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:42,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:44,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:44,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:46,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:46,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:48,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:50,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:50,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:26:51,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:52,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:53,112 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:26:54,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:54,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:26:56,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:57,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:26:58,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:26:59,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:00,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:02,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:02,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:04,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:05,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:06,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:08,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:08,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:09,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:27:10,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:12,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:13,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:14,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:14,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:16,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:18,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:19,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:20,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:20,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:22,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:23,115 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:27:24,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:24,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:27:24,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:26,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:28,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:29,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:30,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:30,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:32,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:34,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:35,532 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:36,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:36,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:38,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:39,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:27:40,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:41,581 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:42,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:44,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:44,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:46,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:47,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:48,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:50,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:50,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:27:52,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:52,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:53,119 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:27:54,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:54,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:27:56,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:58,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:27:58,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:27:59,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:00,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:02,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:03,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:04,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:05,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:06,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:08,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:09,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:09,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:28:10,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:12,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:14,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:14,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:14,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:16,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:18,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:20,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:20,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:20,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:22,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:23,120 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:28:24,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:24,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:28:25,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:26,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:28,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:29,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:30,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:31,423 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:32,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:34,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:35,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:36,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:36,956 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:38,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:39,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:28:40,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:42,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:42,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:44,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:44,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:46,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:48,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:48,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:50,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:50,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:52,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:53,123 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:28:53,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:54,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:54,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:28:56,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:58,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:28:59,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:28:59,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,352 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,450 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,541 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,542 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,591 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,632 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:28:59,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:28:59,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:29:00,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:29:00,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:29:00,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:29:00,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:02,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:04,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:04,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:05,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:06,486 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:08,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:09,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:29:10,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:10,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:12,493 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:14,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:14,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:15,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:16,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:18,500 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:20,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:20,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:21,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:22,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:23,126 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:29:24,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:24,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:29:26,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:26,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:28,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:29,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:30,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:32,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:32,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:34,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:35,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:36,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:37,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:38,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:39,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:29:40,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:42,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:43,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:44,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:44,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:46,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:48,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:49,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:50,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:50,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:29:52,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:53,130 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:29:54,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:54,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:29:54,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:29:56,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:58,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:29:59,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:00,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:00,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:02,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:04,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:05,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:05,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:06,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:08,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:09,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:30:10,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:11,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:12,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:14,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:14,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:16,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:16,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:18,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:20,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:20,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:22,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:22,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:23,132 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:30:24,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:24,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:30:26,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:27,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:28,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:29,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:30,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:32,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:33,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:34,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:35,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:36,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:38,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:38,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:39,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:30:40,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:42,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:44,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:44,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:44,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:46,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:48,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:50,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:50,561 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:30:50,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:52,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:53,135 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:30:54,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:54,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:30:55,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:30:56,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:58,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:30:59,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:00,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:01,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:02,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:04,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:05,563 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:06,623 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:06,627 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:08,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:09,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:31:10,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:12,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:12,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:14,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:14,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:16,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:17,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:18,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:20,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:20,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:22,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:23,138 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:31:23,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:24,647 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:24,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:31:26,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:28,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:28,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:29,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:30,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:32,657 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:34,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:34,659 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:35,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:36,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:38,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:39,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:31:39,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:40,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:42,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:44,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:44,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:45,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:46,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:48,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:50,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:31:50,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:50,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:52,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:53,143 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:31:54,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:31:54,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:56,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:31:56,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:58,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:31:59,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:00,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:01,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:02,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:04,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:05,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:06,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:07,518 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:08,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:09,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:32:10,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:12,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:13,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:14,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:14,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:16,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:18,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:18,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:20,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:20,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:22,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:23,143 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:32:24,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:24,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:32:24,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:26,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:28,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:29,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:29,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:30,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:32,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:34,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:35,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:35,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:36,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:38,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:39,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:32:40,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:40,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:42,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:44,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:44,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:46,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:46,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:48,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:50,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:32:50,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:51,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:52,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:53,146 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:32:54,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:32:54,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:56,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:57,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:32:58,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:32:59,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:00,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:02,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:03,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:04,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:05,582 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:06,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:08,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:08,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:09,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:33:10,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:12,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:14,104 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:14,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:14,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:16,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:18,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:19,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:20,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:20,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:22,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:23,149 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:33:24,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:33:24,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:25,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:26,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:28,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:29,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:30,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:30,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:32,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:34,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:35,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:36,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:36,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:38,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:39,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:33:40,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:41,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:42,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:44,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:44,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:46,810 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:47,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:48,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:50,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:33:50,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:52,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:52,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:53,154 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:33:54,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:33:54,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:56,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:58,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:33:58,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:33:59,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:00,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:02,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:03,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:04,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:05,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:06,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:08,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:09,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:09,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:34:10,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:12,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:14,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:14,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:15,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:16,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:18,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:20,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:20,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:20,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:22,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:23,154 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:34:24,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:34:24,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:26,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:26,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:28,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:29,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:30,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:31,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:32,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:34,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:35,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:36,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:37,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:38,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:39,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:34:40,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:42,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:42,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:44,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:45,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:46,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:48,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:48,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:50,598 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:34:50,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:52,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:53,158 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:34:54,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:54,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:34:54,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:56,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:58,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:34:59,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:34:59,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:00,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:02,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:04,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:04,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:05,600 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:06,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:08,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:09,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:35:10,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:10,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:12,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:14,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:14,914 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:15,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:16,916 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:18,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:20,603 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:20,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:21,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:22,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:23,161 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:35:24,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:35:24,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:26,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:26,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:28,930 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:29,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:30,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:32,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:32,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:34,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:35,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:36,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:38,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:38,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:39,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:35:40,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:42,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:43,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:44,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:44,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:46,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:48,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:49,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:50,607 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:35:50,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:52,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:53,164 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:35:54,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:35:54,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:35:54,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:56,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:58,963 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:35:59,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:00,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:00,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:02,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:04,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:05,609 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:05,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:06,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:08,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:09,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:36:10,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:11,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:12,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:14,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:14,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:16,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:16,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:18,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:20,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:20,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:22,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:22,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:23,166 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:36:24,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:36:24,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:26,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:27,891 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:28,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:29,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:30,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:33,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:33,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:35,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:35,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:37,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:38,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:39,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:39,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:36:41,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:43,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:44,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:44,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:45,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:47,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:49,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:50,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:50,616 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:36:51,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:53,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:53,169 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:36:54,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:36:55,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:55,574 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:36:57,029 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:59,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:36:59,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:01,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:01,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:03,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:05,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:05,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:06,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:07,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:09,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:09,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:37:11,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:12,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:13,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:14,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:15,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:17,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:17,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:19,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:20,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:21,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:23,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:23,172 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:37:23,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:24,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:37:25,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:27,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:28,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:29,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:29,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:31,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:33,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:34,284 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:35,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:35,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:37,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:39,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:39,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:37:39,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:41,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:43,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:44,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:45,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:45,357 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:47,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:49,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:50,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:37:50,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:51,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:53,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:53,175 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:37:54,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:37:55,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:56,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:37:57,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:59,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:37:59,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:01,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:01,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:03,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:05,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:05,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:07,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:07,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:09,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:09,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:38:11,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:13,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:13,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:14,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:15,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:17,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:18,564 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:19,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:20,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:21,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:23,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:23,179 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:38:24,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:24,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:38:25,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:27,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:29,133 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:29,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:29,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:31,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:33,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:35,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:35,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:35,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:37,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:39,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:39,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:38:40,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:41,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:43,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:44,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:45,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:46,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:47,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:49,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:50,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:38:51,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:51,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:53,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:53,182 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:38:54,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:38:55,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:57,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:57,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:38:59,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:38:59,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:01,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:02,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:03,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:05,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:05,635 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:07,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:08,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:09,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:09,654 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:39:11,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:13,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:13,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:14,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:15,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:17,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:19,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:19,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:20,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:21,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:23,186 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:39:23,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:24,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:39:25,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:25,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:27,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:29,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:29,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:30,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:31,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:33,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:35,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:35,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:36,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:37,210 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:39,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:39,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:39:41,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:41,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:43,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:44,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:45,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:47,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:47,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:49,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:50,642 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:39:51,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:52,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:53,187 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:39:53,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:54,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:39:55,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:57,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:58,291 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:39:59,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:39:59,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:01,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:03,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:03,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:05,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:05,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:07,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:09,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:09,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:09,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:40:11,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:13,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:14,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:14,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:15,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:17,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:19,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:20,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:20,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:21,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:23,190 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:40:23,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:24,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:40:25,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:25,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:27,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:29,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:29,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:31,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:31,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:33,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:35,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:35,649 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:37,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:37,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:39,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:39,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:40:41,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:42,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:43,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:44,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:45,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:47,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:48,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:49,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:50,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:40:51,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:53,193 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:40:53,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:53,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:54,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:40:55,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:57,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:59,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:40:59,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:40:59,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:01,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:03,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:04,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:05,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:05,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:07,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:09,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:09,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:41:10,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:11,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:13,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:14,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:15,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:15,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:17,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:19,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:20,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:21,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:21,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:23,198 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:41:23,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:24,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:41:25,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:26,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:27,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:29,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:29,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:31,344 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:32,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:33,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:34,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:35,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:37,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:37,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:39,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:39,655 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:41:41,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:43,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:43,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:44,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:45,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:47,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:49,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:49,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:49,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:41:51,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:53,198 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:41:53,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:54,538 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:41:54,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:41:55,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:57,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:59,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:41:59,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:00,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:01,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:03,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:04,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:05,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:05,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:07,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:09,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:09,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:42:11,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:11,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:13,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:14,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:15,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:16,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:17,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:19,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:19,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:21,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:22,241 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:23,201 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:42:23,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:24,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:42:25,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:27,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:27,785 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:29,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:29,666 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:31,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:33,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:33,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:34,667 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:35,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:37,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:38,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:39,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:39,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:42:41,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:43,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:44,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:44,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:45,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:47,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:49,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:49,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:42:49,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:51,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:53,204 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:42:53,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:54,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:42:55,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:55,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:42:57,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:59,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:42:59,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:00,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:01,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:03,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:04,672 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:05,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:06,520 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:07,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:09,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:09,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:43:11,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:12,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:13,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:14,674 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:15,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:17,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:17,580 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:19,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:19,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:21,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:23,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:23,208 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:43:23,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:24,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:43:25,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:27,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:28,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:29,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:29,676 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:31,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:33,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:34,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:34,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:35,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:37,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:39,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:39,656 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:43:39,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:41,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:43,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:44,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:44,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:45,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:45,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:47,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:49,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:49,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:43:50,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:51,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:53,209 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:43:53,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:54,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:43:55,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:56,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:43:57,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:59,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:43:59,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:01,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:01,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:03,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:04,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:05,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:07,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:07,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:09,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:09,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:44:11,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:12,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:13,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:14,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:15,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:17,537 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:18,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:19,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:19,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:21,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:23,212 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:44:23,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:24,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:24,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:44:25,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:27,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:29,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:29,638 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:29,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:31,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:33,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:34,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:35,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:35,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:37,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:39,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:39,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:44:40,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:41,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:43,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:44,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:45,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:46,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:47,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:49,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:49,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:44:51,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:51,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:53,215 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:44:53,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:54,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:44:55,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:57,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:44:57,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:59,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:44:59,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:01,589 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:02,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:03,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:04,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:05,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:07,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:08,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:09,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:09,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:45:11,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:13,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:13,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:14,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:15,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:17,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:19,494 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:19,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:19,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:21,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:23,218 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:45:23,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:24,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:45:25,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:25,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:27,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:29,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:29,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:30,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:31,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:33,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:34,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:35,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:36,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:37,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:39,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:39,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:45:41,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:41,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:43,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:44,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:45,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:47,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:47,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:49,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:49,696 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:45:51,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:52,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:53,221 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:45:53,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:54,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:45:55,652 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:57,655 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:58,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:45:59,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:45:59,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:01,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:03,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:03,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:04,698 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:05,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:07,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:09,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:09,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:46:09,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:11,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:13,690 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:14,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:14,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:15,692 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:17,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:19,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:19,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:20,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:21,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:23,223 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:46:23,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:24,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:46:25,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:25,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:27,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:29,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:29,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:31,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:31,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:33,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:34,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:35,715 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:37,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:37,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:39,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:46:39,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:41,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:42,555 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:43,724 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:44,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:45,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:47,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:48,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:49,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:49,731 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:51,733 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:53,226 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:46:53,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:53,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:54,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:46:55,738 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:57,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:46:59,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:46:59,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:46:59,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:01,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:03,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:04,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:04,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:05,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:07,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:09,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:47:09,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:10,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:11,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:13,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:14,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:15,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:15,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:17,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:19,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:19,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:21,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:21,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:23,229 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:47:23,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:24,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:47:25,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:26,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:27,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:29,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:29,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:31,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:32,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:33,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:34,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:35,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:37,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:37,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:39,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:47:39,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:41,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:43,466 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:43,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:44,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:45,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:47,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:49,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:49,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:49,797 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:51,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:53,232 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:47:53,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:54,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:47:54,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:47:55,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:57,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:47:59,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:47:59,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:00,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:01,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:03,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:04,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:05,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:05,815 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:07,817 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:09,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:48:09,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:11,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:11,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:13,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:14,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:15,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:16,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:17,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:19,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:19,831 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:21,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:22,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:23,236 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:48:23,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:24,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:48:25,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:27,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:27,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:29,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:29,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:31,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:33,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:33,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:34,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:35,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:37,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:38,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:39,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:48:39,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:41,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:43,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:44,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:44,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:45,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:47,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:49,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:49,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:49,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:51,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:53,241 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:48:53,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:54,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:48:55,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:48:55,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:57,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:48:59,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:48:59,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:00,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:01,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:03,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:04,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:05,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:06,513 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:07,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:09,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:49:09,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:11,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:12,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:13,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:14,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:15,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:17,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:17,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:19,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:19,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:21,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:23,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:23,241 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:49:23,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:24,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:49:25,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:27,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:28,651 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:29,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:29,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:31,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:33,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:34,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:34,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:36,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:37,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:39,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:49:39,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:39,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:41,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:43,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:44,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:45,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:45,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:47,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:49,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:49,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:50,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:51,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:53,243 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:49:53,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:54,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:49:55,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:56,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:49:57,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:49:59,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:49:59,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:01,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:01,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:03,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:04,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:06,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:07,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:07,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:09,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:50:09,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:11,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:12,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:13,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:14,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:15,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:17,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:18,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:19,737 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:19,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:21,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:23,246 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:50:23,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:24,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:24,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:50:25,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:27,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:29,554 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:29,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:29,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:31,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:33,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:34,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:35,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:35,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:37,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:39,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:50:39,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:40,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:41,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:43,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:44,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:46,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:46,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:48,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:49,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:50:50,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:51,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:52,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:53,276 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:50:54,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:54,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:50:56,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:57,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:50:58,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:50:59,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:00,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:02,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:02,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:04,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:04,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:06,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:08,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:09,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:09,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:51:11,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:12,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:13,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:14,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:14,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:16,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:18,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:19,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:19,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:20,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:22,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:23,276 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:51:24,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:24,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:51:24,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:26,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:28,051 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:29,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:30,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:30,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:32,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:34,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:34,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:36,018 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:36,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:39,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:39,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:51:41,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:41,552 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:42,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:44,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:44,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:46,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:47,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:48,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:49,752 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:51:50,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:52,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:52,631 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:53,279 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:51:54,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:54,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:51:56,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:58,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:51:58,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:51:59,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:00,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:02,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:03,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:03,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,912 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:03,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,090 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:04,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,296 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,306 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 09:52:04,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:06,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:08,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:09,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:09,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:52:10,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:12,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:14,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:14,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:14,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:16,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:18,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:19,756 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:20,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:20,440 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:22,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:23,282 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:52:24,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:24,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:52:25,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:26,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:28,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:29,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:30,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:31,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:32,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:34,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:34,758 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:36,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:37,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:38,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:39,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:52:40,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:42,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:42,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:44,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:44,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:46,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:48,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:48,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:49,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:52:50,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:52,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:53,285 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:52:53,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:54,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:54,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:52:56,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:58,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:52:59,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:52:59,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:00,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:02,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:04,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:04,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:04,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:06,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:08,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:09,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:53:10,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:10,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:12,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:14,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:14,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:15,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:16,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:18,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:19,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:20,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:21,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:22,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:23,288 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:53:24,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:24,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:53:26,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:26,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:28,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:29,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:30,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:32,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:32,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:34,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:34,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:36,205 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:37,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:38,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:39,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:53:40,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:42,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:43,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:44,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:44,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:46,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:48,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:49,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:49,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:53:50,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:52,224 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:53,291 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:53:54,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:54,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:53:54,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:53:56,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:58,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:53:59,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:00,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:00,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:02,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:04,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:04,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:05,653 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:06,241 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:08,243 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:09,661 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:54:10,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:11,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:12,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:14,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:14,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:16,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:16,739 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:18,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:19,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:20,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:22,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:22,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:23,296 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:54:24,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:24,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:54:26,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:27,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:28,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:29,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:30,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:32,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:33,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:34,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:34,776 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:36,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:38,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:38,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:39,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:54:40,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:42,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:44,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:44,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:44,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:46,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:48,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:49,778 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:54:49,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:50,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:52,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:53,297 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:54:54,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:54,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:54:55,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:54:56,303 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:58,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:54:59,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:00,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:01,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:02,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:04,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:04,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:06,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:06,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:08,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:09,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:55:10,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:12,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:12,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:14,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:14,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:16,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:17,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:18,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:19,782 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:20,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:22,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:23,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:23,300 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:55:24,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:24,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:55:26,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:28,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:28,699 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:29,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:30,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:32,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:34,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:34,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:34,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:36,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:38,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:39,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:55:39,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:40,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:42,354 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:44,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:44,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:45,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:46,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:48,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:49,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:55:50,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:50,877 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:52,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:53,303 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:55:54,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:54,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:55:56,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:56,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:55:58,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:55:59,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:00,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:01,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:02,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:04,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:04,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:06,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:07,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:08,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:09,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:56:10,386 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:12,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:13,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:14,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:14,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:16,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:18,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:18,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:19,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:20,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:22,399 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:23,308 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:56:24,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:24,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:24,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:56:26,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:28,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:29,679 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:29,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:30,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:32,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:34,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:34,793 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:35,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:36,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:38,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:39,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:56:40,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:40,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:42,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:44,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:44,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:46,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:46,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:48,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:49,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:56:50,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:51,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:52,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:53,308 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:56:54,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:54,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:56:56,438 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:57,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:56:58,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:56:59,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:00,443 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:02,445 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:02,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:04,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:04,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:06,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:08,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:08,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:09,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:57:10,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:12,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:13,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:14,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:14,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:16,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:18,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:19,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:19,800 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:20,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:22,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:23,313 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:57:24,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:24,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:57:25,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:27,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:28,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:29,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:30,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:30,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:32,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:34,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:34,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:36,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:36,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:38,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:39,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:57:40,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:41,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:42,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:44,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:44,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:46,498 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:47,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:48,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:49,805 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:57:50,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:52,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:52,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:53,316 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:57:54,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:54,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:57:56,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:58,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:57:58,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:57:59,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:00,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:03,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:03,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:04,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:04,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:06,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:08,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:09,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:09,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:58:10,527 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:12,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:14,532 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:14,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:14,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:16,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:18,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:19,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:20,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:20,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:22,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:23,321 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:58:24,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:24,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:58:25,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:26,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:28,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:29,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:30,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:31,547 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:32,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:34,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:34,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:36,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:37,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:38,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:39,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:58:40,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:42,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:42,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:44,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:44,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:46,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:48,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:48,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:49,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:58:50,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:52,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:53,321 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:58:53,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:54,579 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:54,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:58:56,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:58,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:58:59,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:58:59,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:00,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:02,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:04,591 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:04,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:04,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:06,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:08,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:09,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:59:10,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:10,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:12,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:14,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:14,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:15,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:16,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:18,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:19,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:20,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:21,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:22,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:23,324 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:59:24,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:24,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:59:26,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:26,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:28,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:29,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:30,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:32,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:32,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:34,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:34,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:36,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:37,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:38,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:39,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:59:40,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:42,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:43,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:44,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:44,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:46,645 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:48,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:49,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:49,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 09:59:50,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:52,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:53,326 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 09:59:54,588 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 09:59:54,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:54,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 09:59:56,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:58,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 09:59:59,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:00,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:00,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:02,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:04,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:04,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:05,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:06,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:08,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:09,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:00:10,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:11,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:12,678 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:14,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:14,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:16,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:16,740 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:18,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:19,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:20,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:22,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:22,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:23,330 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:00:24,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:00:24,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:26,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:27,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:28,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:29,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:30,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:32,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:33,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:34,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:35,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:36,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:38,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:38,708 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:39,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:00:40,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:42,713 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:43,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:44,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:44,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:46,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:48,720 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:49,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:50,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:00:50,722 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:52,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:53,333 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:00:54,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:54,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:00:54,727 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:56,729 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:58,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:00:59,611 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:00:59,686 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:00,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:02,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:04,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:04,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:05,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:07,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:09,665 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:01:09,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:10,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:10,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:13,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:01:13,608 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 10:01:13,615 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 10:01:13,625 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 10:01:13,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 10:01:13,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:14,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:15,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:15,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:15,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:20,627 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:21,620 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:23,336 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:01:25,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:01:26,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:26,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:30,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:30,794 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:31,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:35,831 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:36,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:37,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:01:37,117 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:01:37,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 10:01:37,408 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 10:01:37,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:01:37,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_18_b125d777d8ca0453b1d6.table.json 2025-09-18 10:01:37,824 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 10:01:38,365 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/b7486bnn-media/table/rollouts_18_b125d777d8ca0453b1d6.table.json 2025-09-18 10:01:38,404 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmp42ajrmm1 2025-09-18 10:01:38,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:39,076 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2MzM0NzQ4MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MzE1MzA5NQ==', 'versionIndex': 16}}} 2025-09-18 10:01:39,076 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 10:01:39,077 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_18_b125d777d8ca0453b1d6.table.json with policy now 2025-09-18 10:01:39,409 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/63um6amo-media/table/rollouts_18_b125d777d8ca0453b1d6.table.json 2025-09-18 10:01:40,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:01:40,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:42,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:42,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:44,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:45,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:46,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:47,639 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:48,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:50,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:01:50,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:52,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:53,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:53,339 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:01:54,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:55,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:01:56,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:01:58,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:01:58,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:00,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:00,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:02,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:04,217 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:04,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:05,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:06,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:08,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:09,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:10,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:02:10,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:12,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:14,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:15,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:15,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:16,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:18,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:20,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:20,801 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:20,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:22,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:23,343 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:02:24,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:25,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:02:26,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:26,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:28,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:30,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:30,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:31,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:32,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:34,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:35,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:36,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:37,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:38,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:40,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:02:40,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:42,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:42,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:44,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:45,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:46,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:48,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:48,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:50,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:02:50,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:52,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:53,344 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:02:54,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:02:54,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:55,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:02:56,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:58,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:02:59,537 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:00,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:00,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:02,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:04,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:05,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:05,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:06,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:08,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:10,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:03:10,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:10,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:12,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:14,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:15,093 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:16,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:16,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:18,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:20,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:20,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:21,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:22,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:23,347 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:03:24,951 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:25,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:03:26,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:27,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:28,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:30,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:30,958 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:32,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:32,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:34,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:35,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:36,965 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:38,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:38,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:40,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:03:40,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:42,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:43,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:44,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:45,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:46,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:48,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:49,435 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:50,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:03:50,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:52,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:53,350 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:03:54,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:03:54,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:55,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:03:56,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:03:58,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:00,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:00,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:00,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:02,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:04,998 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:05,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:06,026 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:07,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:09,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:10,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:04:11,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:11,565 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:13,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:15,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:15,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:17,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:17,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:19,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:20,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:21,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:22,648 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:23,020 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:23,354 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:04:25,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:25,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:04:27,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:28,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:29,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:30,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:31,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:33,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:33,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:35,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:35,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:37,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:39,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:39,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:40,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:04:41,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:43,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:44,790 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:45,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:45,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:47,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:49,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:50,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:04:50,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:51,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:53,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:53,355 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:04:55,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:04:55,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:55,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:04:57,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:04:59,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:00,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:01,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:01,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:03,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:05,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:05,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:06,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:07,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:09,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:10,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:05:11,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:12,463 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:13,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:15,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:15,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:17,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:17,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:19,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:20,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:21,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:23,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:23,358 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:05:23,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:25,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:05:25,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:27,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:29,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:29,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:30,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:31,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:33,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:34,628 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:35,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:35,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:37,110 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:39,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:40,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:05:40,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:41,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:43,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:45,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:45,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:45,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:47,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:49,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:50,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:05:51,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:51,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:53,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:53,361 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:05:55,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:05:55,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:56,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:05:57,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:05:59,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:00,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:01,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:02,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:03,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:05,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:05,145 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:07,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:07,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:09,150 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:10,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:06:11,152 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:13,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:13,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:15,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:15,157 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:17,159 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:18,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:19,162 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:20,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:21,164 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:23,166 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:23,366 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:06:24,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:25,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:06:25,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:27,171 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:29,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:30,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:30,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:31,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:33,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:35,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:35,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:35,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:37,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:39,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:40,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:06:41,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:41,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:43,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:45,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:45,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:46,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:47,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:49,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:50,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:06:51,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:52,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:53,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:53,366 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:06:55,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:06:55,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:57,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:06:57,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:06:59,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:00,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:01,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:03,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:03,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:05,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:05,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:07,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:08,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:09,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:10,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:07:11,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:13,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:14,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:15,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:15,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:17,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:19,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:19,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:20,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:21,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:23,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:23,369 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:07:25,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:07:25,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:25,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:27,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:29,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:30,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:30,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:31,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:33,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:35,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:35,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:36,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:37,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:39,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:40,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:07:41,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:42,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:43,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:45,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:45,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:47,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:47,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:49,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:50,138 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:07:51,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:53,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:53,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:53,372 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:07:55,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:07:55,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:57,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:07:58,608 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:07:59,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:00,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:01,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:03,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:04,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:05,141 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:05,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:07,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:09,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:09,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:10,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:08:11,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:13,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:15,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:15,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:15,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:17,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:19,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:20,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:20,755 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:21,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:23,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:23,378 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:08:25,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:08:25,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:26,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:27,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:29,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:30,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:31,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:31,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:33,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:35,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:35,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:37,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:37,387 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:39,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:40,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:08:41,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:42,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:43,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:45,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:45,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:47,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:48,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:49,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:50,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:08:51,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:53,342 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:53,378 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:08:54,380 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:08:55,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:08:55,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:57,347 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:59,349 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:08:59,568 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:00,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:01,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:03,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:05,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:05,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:05,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:07,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:09,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:10,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:09:10,664 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:11,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:13,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:15,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:15,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:16,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:17,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:19,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:20,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:21,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:21,747 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:23,380 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:09:23,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:25,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:09:25,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:27,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:27,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:29,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:30,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:31,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:32,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:33,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:35,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:35,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:37,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:38,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:39,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:40,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:09:41,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:43,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:43,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:45,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:45,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:47,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:49,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:49,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:50,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:09:51,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:53,382 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:09:53,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:54,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:09:55,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:09:55,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:57,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:09:59,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:00,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:00,521 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:01,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:03,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:05,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:05,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:06,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:07,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:09,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:10,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:10:11,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:11,599 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:13,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:15,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:15,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:17,133 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:17,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:19,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:20,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:21,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:22,669 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:23,387 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:10:23,450 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:25,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:10:25,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:27,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:28,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:29,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:30,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:31,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:33,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:33,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:35,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:35,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:37,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:39,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:39,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:40,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:10:41,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:43,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:44,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:45,169 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:45,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:47,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:49,481 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:50,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:10:50,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:51,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:53,390 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:10:53,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:55,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:10:55,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:55,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:10:57,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:10:59,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:00,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:01,430 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:01,495 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:03,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:05,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:05,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:06,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:07,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:09,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:10,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:11:11,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:12,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:13,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:15,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:15,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:17,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:18,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:19,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:20,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:21,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:23,392 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:11:23,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:23,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:25,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:11:25,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:27,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:29,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:29,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:30,177 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:31,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:33,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:34,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:35,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:35,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:37,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:39,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:40,031 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:11:40,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:41,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:43,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:45,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:45,544 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:45,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:47,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:49,549 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:50,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:11:51,256 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:51,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:53,395 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:11:53,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:55,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:11:55,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:56,797 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:11:57,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:11:59,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:00,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:01,563 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:02,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:03,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:05,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:05,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:07,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:07,869 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:09,573 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:10,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:12:11,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:13,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:13,577 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:15,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:15,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:17,582 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:18,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:19,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:20,185 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:21,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:23,398 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:12:23,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:24,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:25,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:12:25,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:27,596 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:29,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:30,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:30,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:31,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:33,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:35,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:35,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:35,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:37,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:39,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:40,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:12:41,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:41,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:43,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:45,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:45,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:46,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:47,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:49,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:50,190 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:12:51,625 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:52,194 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:53,401 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:12:53,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:55,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:12:55,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:57,632 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:12:57,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:12:59,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:00,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:01,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:03,275 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:03,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:05,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:05,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:07,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:08,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:09,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:10,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:13:11,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:13,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:14,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:15,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:15,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:17,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:19,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:19,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:20,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:21,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:23,404 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:13:23,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:25,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:13:25,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:25,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:27,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:29,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:30,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:30,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:31,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:33,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:35,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:35,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:36,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:37,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:39,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:40,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:13:41,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:42,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:43,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:45,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:45,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:47,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:47,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:49,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:50,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:13:51,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:53,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:53,407 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:13:53,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:55,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:13:55,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:57,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:13:58,641 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:13:59,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:00,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:01,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:03,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:04,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:05,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:05,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:07,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:09,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:09,717 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:10,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:14:11,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:13,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:15,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:15,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:15,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:17,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:19,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:20,206 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:20,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:21,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:23,409 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:14:23,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:25,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:14:25,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:26,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:27,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:29,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:30,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:31,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:31,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:33,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:35,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:35,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:37,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:37,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:39,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:40,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:14:41,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:42,925 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:43,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:45,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:45,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:47,760 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:48,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:49,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:50,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:14:51,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:53,412 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:14:53,767 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:54,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:55,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:14:55,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:57,772 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:14:59,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:14:59,774 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:00,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:01,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:03,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:05,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:05,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:05,781 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:07,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:09,786 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:10,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:15:10,621 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:11,788 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:13,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:15,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:15,793 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:16,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:17,795 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:19,798 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:20,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:21,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:21,800 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:23,415 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:15:23,802 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:25,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:15:25,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:27,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:27,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:29,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:30,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:31,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:32,798 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:33,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:35,219 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:35,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:37,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:38,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:39,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:40,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:15:41,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:43,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:43,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:45,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:45,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:47,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:49,418 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:49,833 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:50,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:15:51,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:53,421 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:15:53,838 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:54,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:15:55,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:15:55,840 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:57,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:15:59,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:00,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:00,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:01,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:03,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:05,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:05,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:06,028 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:07,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:09,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:10,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:16:11,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:11,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:13,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:15,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:15,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:15,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,920 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:15,969 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,066 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,077 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,113 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,123 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,146 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,298 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,336 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,663 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:16:16,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:17,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:19,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:20,226 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:21,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:21,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:23,420 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:16:23,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:25,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:16:25,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:26,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:27,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:29,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:30,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:31,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:32,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:33,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:35,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:35,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:37,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:38,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:39,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:40,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:16:41,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:43,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:43,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:45,230 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:45,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:47,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:49,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:49,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:50,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:16:51,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:53,423 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:16:53,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:54,626 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:16:55,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:16:55,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:57,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:16:59,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:00,165 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:00,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:01,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:03,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:05,235 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:05,697 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:05,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:07,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:09,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:10,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:17:11,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:11,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:13,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:15,236 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:15,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:16,780 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:17,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:19,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:20,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:21,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:22,316 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:23,426 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:17:23,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:25,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:17:25,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:27,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:27,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:29,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:30,239 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:31,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:33,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:33,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:35,240 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:35,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:37,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:38,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:39,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:40,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:17:41,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:43,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:44,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:45,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:45,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:47,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:49,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:49,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:50,243 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:17:51,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:53,438 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:17:53,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:55,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:17:55,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:17:55,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:57,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:17:59,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:00,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:01,049 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:01,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:03,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:05,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:05,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:06,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:07,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:09,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:10,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:18:11,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:12,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:14,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:15,247 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:16,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:17,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:18,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:20,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:20,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:22,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:23,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:23,431 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:18:24,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:25,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:18:26,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:28,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:28,732 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:30,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:30,249 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:32,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:34,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:34,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:35,250 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:36,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:38,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:39,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:40,032 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:40,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:18:42,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:44,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:45,252 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:45,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:46,039 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:48,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:50,044 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:50,253 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:18:50,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:52,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:53,434 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:18:54,049 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:55,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:18:56,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:18:56,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:18:58,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:00,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:00,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:01,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:02,058 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:04,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:05,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:06,063 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:07,514 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:08,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:10,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:19:10,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:12,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:13,064 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:14,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:15,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:16,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:18,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:18,605 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:20,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:20,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:22,081 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:23,437 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:19:24,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:24,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:25,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:19:26,086 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:28,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:29,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:30,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:30,261 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:32,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:34,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:35,201 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:35,262 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:36,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:38,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:40,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:19:40,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:40,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:42,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:44,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:45,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:46,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:46,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:48,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:50,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:50,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:19:51,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:52,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:53,442 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:19:54,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:55,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:19:56,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:19:57,349 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:19:58,123 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:00,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:00,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:02,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:02,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:04,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:05,267 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:06,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:08,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:08,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:10,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:20:10,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:12,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:13,985 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:14,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:15,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:16,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:18,147 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:19,516 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:20,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:20,270 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:22,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:23,443 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:20:24,154 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:25,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:20:25,052 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:26,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:28,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:30,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:30,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:30,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:32,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:34,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:35,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:36,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:36,167 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:38,169 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:40,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:20:40,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:41,694 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:42,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:44,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:45,273 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:46,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:47,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:48,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:50,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:50,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:20:52,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:52,781 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:20:53,447 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:20:54,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:55,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:20:56,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:58,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:20:58,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:00,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:00,276 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:02,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:03,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:04,200 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:05,277 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:06,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:08,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:09,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:10,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:21:10,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:12,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:14,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:14,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:15,279 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:16,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:18,217 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:20,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:20,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:20,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:22,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:23,450 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:21:24,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:25,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:21:26,023 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:26,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:28,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:30,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:30,282 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:31,562 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:32,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:34,236 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:35,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:36,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:37,109 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:38,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:40,036 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:21:40,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:42,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:42,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:44,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:45,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:46,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:48,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:48,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:50,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:50,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:21:52,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:53,453 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:21:53,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:21:54,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:55,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:21:56,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:58,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:21:59,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:00,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:00,287 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:02,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:04,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:04,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:05,288 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:06,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:08,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:10,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:22:10,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:10,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:12,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:14,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:15,289 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:15,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:16,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:18,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:20,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:20,290 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:21,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:22,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:23,457 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:22:24,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:25,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:22:26,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:26,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:28,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:30,292 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:30,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:32,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:32,483 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:34,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:35,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:36,308 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:38,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:38,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:40,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:22:40,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:42,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:43,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:44,317 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:45,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:46,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:48,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:49,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:50,295 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:22:50,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:52,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:53,459 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:22:54,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:54,640 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:22:55,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:22:56,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:22:58,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:00,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:00,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:00,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:02,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:04,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:05,297 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:05,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:06,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:08,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:10,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:23:10,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:11,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:12,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:14,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:15,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:16,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:16,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:18,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:20,300 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:20,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:22,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:22,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:23,462 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:23:24,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:25,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:23:26,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:27,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:28,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:30,302 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:30,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:32,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:33,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:34,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:35,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:36,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:38,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:38,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:40,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:23:40,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:42,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:44,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:44,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:45,304 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:46,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:48,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:50,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:50,305 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:23:50,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:52,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:53,465 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:23:54,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:55,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:23:55,553 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:23:56,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:23:58,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:00,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:00,404 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:01,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:02,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:04,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:05,307 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:06,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:06,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:08,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:10,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:24:10,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:12,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:12,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:14,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:15,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:16,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:17,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:18,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:20,310 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:20,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:22,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:23,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:23,468 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:24:24,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:25,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:24:26,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:28,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:28,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:30,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:30,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:32,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:34,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:34,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:35,312 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:36,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:38,448 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:39,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:40,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:24:40,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:42,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:44,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:45,314 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:45,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:46,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:48,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:50,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:24:50,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:50,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:52,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:53,470 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:24:54,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:55,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:24:56,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:24:56,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:24:58,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:00,317 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:00,474 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:02,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:02,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:04,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:05,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:06,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:07,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:08,483 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:10,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:25:10,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:12,488 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:13,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:14,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:15,319 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:16,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:18,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:18,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:20,320 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:20,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:22,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:23,474 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:25:24,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:24,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:25,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:25:26,503 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:28,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:29,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:30,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:30,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:32,510 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:34,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:35,214 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:35,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:36,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:38,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:40,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:25:40,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:40,745 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:42,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:44,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:45,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:46,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:46,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:48,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:50,325 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:25:50,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:51,789 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:52,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:53,475 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:25:54,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:55,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:25:56,539 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:25:57,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:25:58,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:00,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:00,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:02,546 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:02,848 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:04,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:05,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:06,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:08,386 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:08,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:10,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:26:10,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:12,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:13,930 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:14,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:15,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:16,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:18,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:19,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:20,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:20,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:22,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:23,478 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:26:24,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:25,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:25,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:26:26,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:28,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:30,332 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:30,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:30,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:32,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:34,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:35,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:36,087 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:36,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:38,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:40,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:26:40,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:41,636 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:42,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:44,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:45,335 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:46,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:47,182 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:48,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:50,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:26:50,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:52,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:52,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:53,481 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:26:54,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:55,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:26:56,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:26:58,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:26:58,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:00,338 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:00,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:02,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:03,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:04,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:05,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:06,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:08,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:09,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:10,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:27:10,639 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:12,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:14,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:14,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:15,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:16,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:18,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:20,341 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:20,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:20,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:22,654 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:23,487 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:27:24,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:25,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:27:25,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:26,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:28,661 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:30,343 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:30,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:31,533 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:32,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:34,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:35,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:36,669 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:37,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:38,671 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:40,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:27:40,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:42,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:42,676 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:44,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:45,346 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:46,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:48,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:48,683 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:50,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:27:50,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:52,688 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:53,487 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:27:53,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:27:54,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:55,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:27:56,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:58,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:27:59,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:00,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:00,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:02,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:04,702 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:04,735 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:05,351 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:06,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:08,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:10,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:28:10,274 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:10,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:12,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:14,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:15,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:15,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:16,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:18,718 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:20,353 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:20,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:21,350 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:22,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:23,491 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:28:24,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:25,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:28:26,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:26,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:28,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:30,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:30,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:32,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:32,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:34,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:35,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:36,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:37,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:38,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:40,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:28:40,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:42,746 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:43,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:44,748 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:45,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:46,750 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:48,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:49,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:50,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:28:50,755 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:52,757 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:53,494 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:28:54,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:28:54,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:55,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:28:56,762 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:28:58,764 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:00,120 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:00,360 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:00,766 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:02,769 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:04,771 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:05,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:05,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:06,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:08,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:10,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:29:10,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:11,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:12,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:14,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:15,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:16,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:16,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:18,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:20,364 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:20,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:22,278 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:22,791 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:23,500 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:29:24,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:25,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:29:26,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:27,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:28,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:30,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:30,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:32,804 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:33,344 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:34,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:35,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:36,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:38,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:38,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:40,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:29:40,813 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:42,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:44,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:44,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:45,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:46,820 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:48,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:49,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:50,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:29:50,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:52,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:53,499 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:29:54,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:55,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:29:55,511 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:29:56,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:29:58,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:00,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:00,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:01,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:02,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:04,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:05,371 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:06,585 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:06,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:08,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:10,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:30:10,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:12,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:12,851 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:14,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:15,372 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:16,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:17,644 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:18,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:20,373 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:20,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:22,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:23,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:23,502 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:30:24,864 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:25,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:30:26,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:28,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:28,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:30,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:30,871 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:32,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:34,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:34,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:35,376 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:36,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:38,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:39,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:40,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:30:40,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:42,886 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:44,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:45,334 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:45,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:46,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:48,893 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:50,378 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:30:50,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:50,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:52,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:53,505 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:30:54,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:55,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:30:56,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:30:56,901 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:30:58,903 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:00,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:00,905 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:01,961 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:02,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:04,910 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:05,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:06,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:07,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:08,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:10,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:31:10,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:12,919 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:13,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:14,921 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:15,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:16,924 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:18,557 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:18,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:20,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:20,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:22,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:23,507 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:31:24,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:24,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:25,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:31:26,936 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:28,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:29,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:30,384 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:30,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:32,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:34,945 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:35,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:35,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:36,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:38,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:40,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:31:40,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:40,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:42,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:44,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:45,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:46,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:46,960 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:48,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:50,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:31:50,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:51,738 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:52,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:53,511 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:31:54,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:55,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:31:56,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:31:57,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:31:58,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:00,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:00,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:02,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:02,978 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:04,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:05,391 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:06,983 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:08,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:08,985 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:10,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:32:10,988 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:12,990 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:13,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:14,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:15,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:16,995 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:18,997 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:19,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:20,394 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:20,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:23,002 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:23,539 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:32:24,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:25,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:25,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:32:27,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:29,008 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:30,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:30,512 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:31,010 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:33,013 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:35,015 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:35,396 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:36,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:37,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:39,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:40,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:32:41,022 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:41,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:43,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:45,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:45,398 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:47,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:47,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:49,031 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:50,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:32:51,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:52,693 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:53,036 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:53,546 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:32:55,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:55,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:32:57,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:32:58,231 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:32:59,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:00,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:01,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:03,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:03,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:05,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:05,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:07,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:09,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:09,339 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:10,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:33:11,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:13,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:14,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:15,061 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:15,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:17,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:19,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:20,404 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:20,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:21,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:23,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:23,544 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:33:25,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:33:25,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:25,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:27,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:29,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:30,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:31,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:31,498 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:33,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:35,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:35,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:37,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:37,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:39,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:40,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:33:41,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:42,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:43,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:45,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:45,408 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:47,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:48,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:49,102 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:50,409 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:33:51,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:53,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:53,548 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:33:53,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:33:55,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:33:55,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:57,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:59,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:33:59,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:00,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:01,115 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:03,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:04,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:05,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:05,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:07,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:09,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:10,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:34:10,251 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:11,127 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:13,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:15,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:15,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:15,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:17,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:19,137 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:20,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:21,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:21,318 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:23,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:23,551 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:34:25,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:34:25,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:26,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:27,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:29,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:30,416 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:31,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:32,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:33,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:35,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:35,417 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:37,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:37,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:39,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:40,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:34:41,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:43,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:43,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:45,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:45,419 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:47,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:48,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:49,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:50,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:34:51,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:53,176 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:53,557 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:34:54,558 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:34:55,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:34:55,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:57,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:34:59,183 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:00,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:00,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:01,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:03,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:05,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:05,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:05,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:07,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:09,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:10,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:35:11,136 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:11,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:13,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:15,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:15,424 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:16,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:17,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:19,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:20,425 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:21,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:22,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:23,212 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:23,557 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:35:25,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:35:25,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:27,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:27,743 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:29,219 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:30,426 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:31,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:33,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:33,285 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:35,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:35,427 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:37,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:38,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:39,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:40,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:35:41,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:43,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:44,370 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:45,238 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:45,428 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:47,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:49,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:49,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:50,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:35:51,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:53,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:53,560 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:35:55,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:35:55,249 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:55,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:35:57,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:35:59,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:00,431 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:00,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:01,256 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:03,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:05,261 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:05,432 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:06,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:07,263 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:09,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:10,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:36:11,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:12,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:13,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:15,273 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:15,433 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:17,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:17,606 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:19,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:20,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:21,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:23,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:23,282 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:23,563 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:36:25,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:36:25,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:27,287 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:28,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:29,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:30,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:31,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:33,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:34,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:35,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:35,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:37,299 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:39,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:39,771 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:40,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:36:41,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:43,306 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:45,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:45,321 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:45,438 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:47,311 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:49,313 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:50,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:36:50,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:51,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:53,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:53,566 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:36:55,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:36:55,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:56,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:36:57,322 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:36:59,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:00,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:01,327 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:01,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:03,329 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:05,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:05,442 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:07,334 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:07,441 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:09,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:10,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:37:11,339 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:12,975 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:13,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:15,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:15,443 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:17,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:18,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:19,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:20,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:21,351 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:23,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:23,570 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:37:23,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:25,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:37:25,356 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:27,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:28,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:29,361 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:30,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:31,363 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:33,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:33,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:35,368 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:35,447 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:37,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:39,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:39,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:40,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:37:41,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:43,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:44,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:45,379 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:45,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:47,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:49,384 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:50,315 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:50,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:37:51,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:53,390 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:53,572 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:37:55,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:37:55,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:55,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:37:57,395 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:37:59,397 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:00,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:01,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:01,400 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:03,402 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:05,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:05,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:06,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:07,407 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:09,409 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:10,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:38:11,411 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:12,477 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:13,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:15,416 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:15,453 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:17,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:18,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:19,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:20,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:21,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:23,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:23,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:23,575 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:38:25,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:38:25,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:27,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:29,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:29,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:30,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:31,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:33,436 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:34,643 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:35,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:35,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:37,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:39,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:40,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:38:40,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:41,446 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:43,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:45,451 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:45,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:45,722 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:47,453 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:49,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:50,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:38:51,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:51,458 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:53,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:53,578 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:38:55,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:38:55,463 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:56,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:38:57,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:38:59,468 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:00,461 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:01,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:02,330 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:03,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:05,462 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:05,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:07,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:07,874 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:09,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:10,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:39:11,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:13,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:13,484 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:15,464 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:15,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:17,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:18,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:19,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:20,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:20,524 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,560 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,570 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,575 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,610 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,671 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,677 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,678 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,683 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,684 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,723 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,724 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,727 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,729 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,751 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,773 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,774 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,796 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,814 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,854 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,886 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,897 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:20,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:21,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:21,264 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:21,265 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:39:21,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:23,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:23,580 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:39:24,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:25,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:39:25,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:27,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:29,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:30,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:30,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:31,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:33,508 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:35,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:35,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:35,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:37,513 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:39,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:40,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:39:41,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:41,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:43,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:45,469 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:45,522 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:46,787 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:47,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:49,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:50,471 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:39:51,529 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:52,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:53,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:53,582 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:39:55,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:39:55,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:57,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:39:57,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:39:59,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:00,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:01,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:03,405 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:03,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:05,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:05,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:07,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:08,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:09,551 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:10,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:40:11,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:13,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:14,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:15,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:15,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:17,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:19,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:20,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:20,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:21,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:23,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:23,588 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:40:25,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:40:25,569 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:25,572 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:27,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:29,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:30,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:31,101 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:31,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:33,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:35,479 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:35,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:36,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:37,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:39,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:40,044 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:40:41,475 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:41,587 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:43,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:45,480 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:45,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:46,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:47,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:49,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:50,482 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:40:51,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:52,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:53,589 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:40:53,602 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:55,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:40:55,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:57,374 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:40:57,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:40:59,608 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:00,484 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:01,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:02,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:03,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:05,485 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:05,615 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:07,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:07,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:09,620 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:10,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:41:11,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:13,271 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:13,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:15,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:15,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:17,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:18,566 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:20,487 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:22,544 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:41:22,566 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 10:41:22,574 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 10:41:22,583 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 10:41:22,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 10:41:23,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:23,592 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:41:23,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:26,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:41:28,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:31,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:33,594 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:35,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:37,259 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:39,358 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:41,550 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:41:44,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:41:46,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:41:46,395 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:41:46,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 10:41:46,701 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 10:41:46,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 10:41:47,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:47,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:41:47,666 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_19_ee0d3182ce72199f9003.table.json 2025-09-18 10:41:47,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 10:41:52,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:41:55,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:42:00,812 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:05,813 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:10,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:42:15,815 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:20,816 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:25,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:42:30,817 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:35,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:40,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:42:45,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:50,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:42:55,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:43:00,822 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:05,823 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:10,700 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:43:15,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:20,826 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:25,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:43:30,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:35,828 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:40,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:43:45,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:50,830 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:43:55,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:43:58,130 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/7vu1kkob-media/table/rollouts_19_ee0d3182ce72199f9003.table.json 2025-09-18 10:43:58,188 INFO wandb-upload_1:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpyiavhaxr 2025-09-18 10:43:58,916 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2Mzg2NTY5OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2MzM0NzQ4MQ==', 'versionIndex': 17}}} 2025-09-18 10:43:58,917 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 10:43:58,917 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_19_ee0d3182ce72199f9003.table.json with policy now 2025-09-18 10:43:58,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:43:58,918 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:43:58,919 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:43:58,921 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:43:58,922 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:43:58,925 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:43:59,225 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/o7u4tiiu-media/table/rollouts_19_ee0d3182ce72199f9003.table.json 2025-09-18 10:43:59,822 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:00,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:01,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:03,827 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:04,248 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:05,829 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:05,833 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:07,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:09,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:09,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:10,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:44:11,836 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:13,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:15,322 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:15,835 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:15,841 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:17,843 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:19,846 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:20,837 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:20,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:21,848 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:23,607 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:44:23,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:25,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:44:25,853 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:26,375 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:27,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:29,858 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:30,838 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:31,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:31,910 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:33,862 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:35,839 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:35,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:37,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:37,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:39,869 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:40,701 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:44:41,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:42,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:43,874 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:45,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:45,876 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:47,878 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:48,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:49,881 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:50,842 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:44:51,883 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:53,612 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:44:53,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:53,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:55,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:44:55,888 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:57,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:44:59,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:44:59,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:00,843 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:01,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:03,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:05,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:05,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:05,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:07,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:09,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:10,658 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:10,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:45:11,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:13,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:15,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:15,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:16,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:17,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:19,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:20,847 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:21,725 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:21,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:23,613 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:45:23,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:25,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:45:25,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:27,258 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:27,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:29,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:30,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:31,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:32,799 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:33,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:35,850 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:35,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:37,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:38,340 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:39,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:40,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:45:41,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:43,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:43,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:45,852 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:45,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:47,949 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:49,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:49,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:50,853 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:45:51,954 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:53,616 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:45:53,956 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:54,977 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:45:55,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:45:55,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:57,961 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:45:59,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:00,509 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:00,855 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:01,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:03,968 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:05,856 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:05,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:06,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:07,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:09,975 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:10,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:46:11,602 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:11,977 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:13,980 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:15,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:15,982 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:17,144 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:17,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:19,987 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:20,858 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:21,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:22,689 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:23,619 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:46:23,992 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:25,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:46:25,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:27,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:28,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:29,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:30,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:32,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:33,766 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:34,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:35,861 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:36,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:38,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:39,309 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:40,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:40,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:46:42,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:44,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:44,859 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:45,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:46,018 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:48,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:50,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:50,393 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:50,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:46:52,025 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:53,625 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:46:54,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:55,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:46:55,928 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:46:56,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:46:58,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:00,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:00,864 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:01,468 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:02,037 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:04,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:05,866 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:06,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:07,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:08,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:10,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:10,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:47:12,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:12,551 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:14,053 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:15,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:16,055 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:18,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:18,082 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:20,060 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:20,868 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:22,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:23,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:23,626 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:47:24,065 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:25,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:47:26,068 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:28,070 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:29,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:30,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:30,870 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:32,075 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:34,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:34,668 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:35,871 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:36,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:38,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:40,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:40,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:40,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:47:42,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:44,090 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:45,750 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:45,872 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:46,092 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:48,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:50,097 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:50,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:47:51,283 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:52,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:53,628 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:47:54,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:55,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:47:56,104 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:47:56,818 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:47:58,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:00,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:00,875 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:02,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:02,348 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:04,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:05,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:06,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:07,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:08,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:10,121 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:10,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:48:12,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:13,429 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:14,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:15,878 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:16,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:18,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:18,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:20,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:20,879 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:22,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:23,631 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:48:24,139 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:24,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:25,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:48:26,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:28,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:30,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:30,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:30,881 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:32,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:34,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:35,573 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:35,882 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:36,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:38,155 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:40,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:40,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:48:41,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:42,160 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:44,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:45,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:46,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:46,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:48,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:50,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:50,885 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:48:52,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:52,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:53,634 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:48:54,174 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:55,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:48:56,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:48:57,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:48:58,179 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:00,181 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:00,887 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:02,184 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:03,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:04,186 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:05,888 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:06,188 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:08,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:08,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:10,193 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:10,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:49:12,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:14,198 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:14,329 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:15,889 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:16,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:18,203 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:19,860 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:20,206 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:20,890 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:22,208 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:23,637 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:49:24,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:25,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:25,704 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:49:26,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:28,215 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:30,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:30,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:30,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:32,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:34,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:35,894 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:36,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:36,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:38,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:40,230 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:40,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:49:42,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:42,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:44,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:45,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:46,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:47,569 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:48,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:50,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:50,896 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:49:52,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:53,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:49:53,639 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:49:54,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:55,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:49:56,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:58,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:49:58,629 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:00,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:00,898 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:02,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:04,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:04,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:05,899 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:06,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:08,265 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:09,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:10,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:10,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:50:12,270 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:14,272 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:15,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:15,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:16,275 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:18,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:20,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:20,762 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:20,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:22,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:23,642 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:50:24,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:25,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:50:26,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:26,286 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:28,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:30,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:30,903 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:31,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:32,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:34,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:35,904 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:36,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:37,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:38,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:40,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:40,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:50:42,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:42,895 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:44,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:45,906 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:46,310 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:48,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:48,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:50,315 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:50,907 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:50:52,318 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:53,645 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:50:53,646 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:50:54,320 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:55,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:50:56,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:58,325 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:50:59,497 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:00,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:00,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:02,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:04,332 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:05,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:05,909 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:06,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:08,337 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:10,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:10,579 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:10,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:51:12,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:14,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:15,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:16,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:16,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:18,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:20,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:20,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:21,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:22,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:23,648 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:51:24,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:25,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:51:26,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:27,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:28,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:30,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:30,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:32,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:32,728 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:34,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:35,914 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:36,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:38,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:38,374 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:40,376 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:40,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:51:42,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:43,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:44,381 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:45,916 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:46,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:48,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:49,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:50,388 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:50,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:51:52,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:53,651 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:51:54,393 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:54,893 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:51:55,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:51:56,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:51:58,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:00,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:00,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:00,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:02,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:04,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:05,919 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:05,980 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:06,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:08,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:10,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:10,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:52:11,504 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:12,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:14,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:15,921 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:16,420 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:17,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:18,422 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:20,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:20,922 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:22,427 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:22,571 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:23,654 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:52:24,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:25,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:52:26,432 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:28,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:28,434 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:30,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:30,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:32,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:33,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:34,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:35,924 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:36,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:38,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:39,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:40,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:40,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:52:42,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:44,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:44,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:45,926 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:46,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:48,460 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:50,269 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:50,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:50,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:52:52,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:53,658 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:52:54,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:55,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:52:55,808 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:52:56,470 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:52:58,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:00,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:00,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:01,363 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:02,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:04,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:05,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:06,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:06,901 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:08,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:10,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:10,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:53:12,436 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:12,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:14,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:15,931 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:16,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:17,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:18,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:20,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:20,932 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:22,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:23,503 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:23,660 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:53:24,505 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:25,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:53:26,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:28,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:29,039 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:30,512 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:30,934 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:32,515 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:34,517 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:34,567 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:35,935 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:36,520 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:38,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:40,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:40,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:40,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:53:42,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:44,530 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:45,647 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:45,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:46,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:48,535 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:50,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:50,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:53:51,174 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:52,540 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:53,663 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:53:54,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:55,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:53:56,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:53:56,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:53:58,548 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:00,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:00,939 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:02,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:02,553 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:04,555 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:05,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:06,558 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:07,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:08,560 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:10,562 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:10,706 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:54:12,565 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:13,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:14,567 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:15,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:16,570 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:18,572 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:18,862 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:20,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:20,942 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:22,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:23,668 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:54:24,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:24,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:25,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:54:26,583 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:28,585 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:29,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:30,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:30,944 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:32,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:34,592 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:35,459 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:35,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:36,594 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:38,597 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:40,599 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:40,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:54:40,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:42,601 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:44,604 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:45,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:46,528 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:46,606 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:48,609 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:50,611 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:50,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:54:52,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:52,613 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:53,668 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:54:54,616 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:55,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:54:56,618 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:54:57,593 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:54:58,621 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:00,623 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:00,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:02,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:03,128 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:04,628 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:05,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:06,630 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:08,633 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:08,673 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:10,635 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:10,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:55:12,637 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:14,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:14,640 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:15,952 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:16,642 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:18,644 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:19,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:20,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:20,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:22,649 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:23,671 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:55:24,651 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:25,301 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:25,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:55:26,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:28,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:30,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:30,834 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:30,954 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:32,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:34,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:35,955 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:36,368 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:36,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:38,667 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:40,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:40,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:55:41,913 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:42,672 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:44,674 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:45,957 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:46,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:47,449 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:48,679 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:50,681 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:50,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:55:52,684 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:53,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:53,675 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:55:54,686 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:55,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:55:56,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:55:58,543 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:55:58,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:00,693 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:00,959 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:02,695 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:04,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:04,698 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:05,960 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:06,700 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:08,703 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:09,622 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:10,705 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:10,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:56:12,707 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:14,710 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:15,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:15,962 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:16,712 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:18,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:20,685 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:20,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:20,963 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:22,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:23,680 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:56:24,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:25,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:56:26,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:26,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:28,725 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:30,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:30,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:31,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:32,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:34,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:35,965 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:36,735 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:37,281 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:38,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:40,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:56:40,740 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:42,742 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:42,806 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:44,744 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:45,967 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:46,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:48,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:48,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:50,751 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:50,968 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:56:52,753 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:53,680 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:56:53,682 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:56:54,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:55,708 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:56:56,758 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:58,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:56:59,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:00,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:00,970 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:02,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:04,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:04,941 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:05,972 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:06,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:08,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:10,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:10,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:57:10,775 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:12,777 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:14,779 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:15,973 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:16,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:16,782 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:18,784 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:20,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:20,974 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:21,577 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:22,789 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:23,683 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:57:24,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:25,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:57:26,805 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:27,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:28,807 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:30,809 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:30,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:32,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:32,812 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:34,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:35,976 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:36,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:38,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:38,819 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:40,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:57:40,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:42,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:43,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:44,826 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:45,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:46,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:48,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:49,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:50,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:50,979 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:57:52,835 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:53,686 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:57:54,807 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:57:54,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:55,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:57:56,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:57:58,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:00,347 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:00,845 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:00,981 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:02,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:04,850 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:05,892 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:05,982 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:06,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:08,855 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:10,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:58:10,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:11,437 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:12,860 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:14,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:15,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:16,865 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:16,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:18,867 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:20,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:20,984 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:22,526 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:22,872 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:23,691 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:58:24,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:25,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:58:26,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:28,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:28,879 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:30,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:30,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:32,885 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:33,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:34,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:35,987 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:36,890 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:38,892 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:39,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:40,709 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:58:40,895 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:42,897 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:44,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:44,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:45,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:46,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:48,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:50,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:50,907 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:50,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:58:52,909 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:53,691 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:58:54,912 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:55,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:58:55,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:58:56,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:58:58,917 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:00,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:00,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:01,268 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:02,922 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:04,925 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:05,992 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:06,786 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:06,927 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:08,929 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:10,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:59:10,932 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:12,324 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:12,934 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:14,937 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:15,994 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:16,939 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:17,863 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:18,941 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:20,943 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:20,995 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:22,946 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:23,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:23,694 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:59:24,948 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:25,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:59:26,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:28,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:28,953 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:30,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:30,997 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:32,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:34,481 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:34,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:35,998 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:36,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:38,967 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:40,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:40,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:59:40,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:42,972 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:44,974 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:45,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:45,999 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:46,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:48,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:50,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:51,000 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 10:59:51,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:52,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:53,696 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 10:59:54,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:55,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 10:59:56,614 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 10:59:56,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 10:59:58,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:00,994 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:01,002 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:02,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:02,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:04,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:06,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:07,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:07,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:09,003 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:10,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:00:11,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:13,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:13,222 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:15,011 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:16,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:17,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:18,764 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:19,016 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:21,005 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:21,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:23,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:23,700 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:00:24,687 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:25,023 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:25,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:00:27,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:29,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:29,836 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:31,007 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:31,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:33,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:35,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:35,377 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:36,008 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:37,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:39,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:40,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:00:40,911 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:41,043 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:43,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:45,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:46,009 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:46,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:47,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:49,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:51,010 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:00:51,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:51,978 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:53,056 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:53,702 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:00:55,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:55,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:00:57,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:00:57,507 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:00:59,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:01,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:01,067 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:03,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:03,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:05,072 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:06,013 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:07,074 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:08,584 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:09,077 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:10,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:01:11,079 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:13,082 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:14,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:15,084 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:16,014 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:17,087 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:19,089 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:19,675 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:21,015 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:21,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:23,094 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:23,705 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:01:25,096 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:25,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:25,710 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:01:27,099 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:29,101 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:30,731 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:31,016 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:31,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:33,106 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:35,108 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:36,017 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:36,263 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:37,111 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:39,113 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:40,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:01:41,116 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:41,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:43,118 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:45,120 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:46,019 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:47,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:47,337 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:49,125 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:51,020 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:01:51,128 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:52,876 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:53,130 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:53,708 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:01:55,132 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:55,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:01:57,135 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:01:58,402 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:01:59,138 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:01,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:01,141 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:03,143 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:03,918 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:05,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:06,022 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:07,148 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:09,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:09,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:10,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:02:11,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:13,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:14,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:15,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:16,024 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:17,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:19,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:20,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:21,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:21,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:23,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:23,711 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:02:25,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:25,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:02:26,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:27,172 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:29,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:31,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:31,178 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:31,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:33,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:35,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:36,027 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:37,097 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:37,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:39,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:40,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:02:41,190 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:42,637 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:43,192 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:45,195 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:46,029 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:47,197 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:48,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:49,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:51,030 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:02:51,202 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:53,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:53,714 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:02:53,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:02:55,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:55,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:02:57,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:59,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:02:59,246 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:01,032 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:01,214 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:03,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:04,784 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:05,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:06,033 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:07,221 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:09,223 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:10,333 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:10,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:03:11,226 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:13,228 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:15,231 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:15,865 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:16,034 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:17,233 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:19,235 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:21,035 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:21,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:21,399 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:23,240 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:23,717 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:03:25,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:25,711 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:03:26,936 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:27,245 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:29,247 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:31,037 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:31,250 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:32,473 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:33,252 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:35,254 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:36,038 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:37,257 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:38,011 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:38,209 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,244 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,245 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,255 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,260 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,294 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,354 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,355 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,356 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,362 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,366 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,367 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,400 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,401 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,407 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,410 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,411 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,413 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,434 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,444 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,446 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,451 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,452 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,456 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,478 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,488 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,489 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,490 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,496 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,499 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,500 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,545 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,546 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,589 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,590 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,633 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,634 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,680 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,948 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,950 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:38,951 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:03:39,259 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:40,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:03:41,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:43,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:43,691 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:45,266 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:46,040 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:47,268 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:49,225 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:49,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:51,041 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:03:51,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:53,277 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:53,721 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:03:54,772 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:03:55,279 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:55,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:03:57,281 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:03:59,284 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:00,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:01,042 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:01,286 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:03,289 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:05,291 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:05,846 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:06,043 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:07,293 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:09,295 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:10,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:04:11,298 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:11,392 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:13,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:15,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:16,045 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:16,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:17,305 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:19,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:21,046 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:21,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:22,460 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:23,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:23,723 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:04:25,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:25,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:04:27,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:28,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:29,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:31,047 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:31,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:33,324 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:33,539 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:35,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:36,048 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:37,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:39,074 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:39,331 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:40,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:04:41,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:43,336 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:44,613 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:45,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:46,050 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:47,341 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:49,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:50,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:51,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:04:51,346 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:53,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:53,726 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:04:55,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:55,681 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:04:55,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:04:57,353 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:04:59,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:01,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:01,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:01,358 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:03,360 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:05,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:06,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:06,767 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:07,365 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:09,367 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:10,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:05:11,370 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:12,299 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:13,372 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:15,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:16,055 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:17,377 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:17,844 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:19,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:21,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:21,382 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:23,382 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:23,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:23,728 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:05:25,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:25,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:05:27,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:28,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:29,392 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:31,057 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:31,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:33,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:34,458 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:35,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:36,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:37,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:39,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:40,003 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:40,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:05:41,405 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:43,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:45,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:45,540 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:46,059 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:47,412 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:49,414 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:51,060 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:05:51,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:51,417 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:53,419 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:53,734 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:05:55,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:55,712 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:05:56,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:05:57,424 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:05:59,426 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:01,062 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:01,429 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:02,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:03,431 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:05,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:06,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:07,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:07,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:09,439 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:10,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:06:11,441 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:13,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:13,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:15,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:16,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:17,455 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:18,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:19,457 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:21,065 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:21,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:23,462 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:23,734 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:06:23,736 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:25,465 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:25,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:06:27,467 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:28,795 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:29,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:31,067 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:31,472 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:33,475 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:33,943 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:35,477 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:36,068 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:37,479 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:39,476 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:39,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:40,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:06:41,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:43,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:45,012 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:45,490 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:46,070 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:47,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:49,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:50,549 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:51,071 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:06:51,496 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:53,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:53,737 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:06:55,501 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:55,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:06:56,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:06:57,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:06:59,506 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:01,072 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:01,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:01,624 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:03,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:05,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:06,073 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:07,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:07,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:09,518 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:10,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:07:11,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:12,703 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:13,523 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:15,525 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:16,075 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:17,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:18,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:19,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:21,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:21,534 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:23,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:23,740 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:07:23,741 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:25,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:25,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:07:27,541 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:29,328 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:29,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:31,078 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:31,545 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:33,547 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:34,884 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:35,550 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:36,079 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:37,552 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:39,554 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:40,422 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:40,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:07:41,556 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:43,559 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:45,561 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:45,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:46,080 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:47,564 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:49,566 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:51,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:07:51,495 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:51,568 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:53,571 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:53,746 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:07:55,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:55,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:07:57,021 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:07:57,576 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:07:59,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:01,083 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:01,581 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:02,559 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:03,584 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:05,586 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:06,084 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:07,588 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:08,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:09,590 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:10,713 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:08:11,593 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:13,595 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:13,625 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:15,598 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:16,085 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:17,600 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:19,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:19,603 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:21,086 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:21,605 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:23,607 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:23,747 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:08:24,749 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:25,610 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:25,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:08:27,612 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:29,614 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:30,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:31,088 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:31,617 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:33,619 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:35,622 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:35,754 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:36,089 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:37,624 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:39,626 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:40,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:08:41,313 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:41,629 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:43,631 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:45,634 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:46,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:46,849 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:47,636 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:49,638 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:51,092 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:08:51,641 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:52,383 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:53,643 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:53,750 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:08:55,646 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:55,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:08:57,648 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:08:57,929 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:08:59,650 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:01,094 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:01,653 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:03,472 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:03,656 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:05,658 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:06,095 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:07,660 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:09,025 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:09,663 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:10,714 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:09:11,665 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:13,668 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:14,576 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:15,670 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:16,096 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:17,673 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:19,675 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:20,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:21,098 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:21,677 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:23,680 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:23,753 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:09:25,659 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:25,682 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:25,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:09:27,685 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:29,687 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:31,099 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:31,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:31,689 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:33,691 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:35,694 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:36,100 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:36,742 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:37,696 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:39,699 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:40,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:09:41,701 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:42,293 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:43,704 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:45,706 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:46,102 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:47,709 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:47,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:49,711 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:51,103 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:09:51,714 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:53,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:53,716 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:53,755 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:09:55,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:09:55,719 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:57,721 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:09:58,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:09:59,723 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:01,105 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:01,726 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:03,728 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:04,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:05,730 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:06,106 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:07,732 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:09,734 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:09,990 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:10,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:10:11,737 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:13,739 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:15,535 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:15,741 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:16,107 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:17,743 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:19,745 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:21,081 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:21,108 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:21,747 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:23,749 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:23,758 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:10:25,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:10:25,752 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:26,618 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:27,754 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:29,756 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:31,110 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:31,759 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:32,152 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:33,761 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:35,763 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:36,111 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:37,670 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:37,765 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:39,768 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:40,715 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:10:41,770 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:43,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:43,773 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:45,776 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:46,112 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:47,778 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:48,761 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:49,780 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:51,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:10:51,783 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:53,761 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:10:53,763 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:53,785 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:55,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:10:55,787 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:57,790 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:10:58,841 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:10:59,792 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:01,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:01,794 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:03,796 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:03,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:05,799 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:06,116 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:07,801 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:09,530 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:09,803 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:10,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:11:11,806 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:13,808 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:15,091 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:15,811 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:16,118 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:17,814 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:19,816 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:20,619 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:21,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:21,818 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:23,764 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:11:23,821 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:25,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:11:25,823 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:26,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:27,825 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:29,828 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:31,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:31,705 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:31,830 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:33,832 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:35,834 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:36,122 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:37,237 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:37,837 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:39,839 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:40,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:11:41,842 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:42,777 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:43,844 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:45,847 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:46,124 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:47,849 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:48,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:49,852 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:51,125 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:11:51,854 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:53,767 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:11:53,769 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:53,857 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:55,716 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:11:55,859 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:57,861 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:11:59,381 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:11:59,863 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:01,126 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:01,866 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:03,868 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:04,923 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:05,870 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:06,127 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:07,873 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:09,875 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:10,455 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:10,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:12:11,877 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:13,880 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:15,882 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:15,991 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:16,129 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:17,884 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:19,887 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:21,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:21,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:21,889 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:23,770 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:12:23,891 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:25,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:12:25,894 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:27,058 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:27,896 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:29,899 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:31,131 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:31,902 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:32,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:33,904 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:35,906 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:36,132 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:37,908 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:38,121 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:39,911 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:40,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:12:41,913 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:43,657 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:43,915 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:45,918 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:46,134 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:47,920 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:49,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:49,923 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:51,135 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:12:51,926 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:53,773 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:12:53,928 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:54,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:12:55,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:12:55,931 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:57,933 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:12:59,935 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:00,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:01,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:01,938 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:03,940 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:05,803 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:05,942 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:06,137 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:07,944 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:09,947 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:10,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:13:11,359 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:11,950 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:13,952 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:15,955 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:16,139 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:16,900 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:17,957 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:19,959 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:21,140 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:21,962 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:22,448 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:23,778 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:13:23,964 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:25,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:13:25,966 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:27,969 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:27,988 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:29,971 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:31,142 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:31,973 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:33,525 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:33,976 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:35,979 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:36,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:37,981 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:39,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:39,984 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:40,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:13:41,986 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:43,989 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:44,586 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:45,991 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:46,145 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:47,993 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:49,996 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:50,119 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:51,147 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:13:51,999 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:53,780 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:13:54,001 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:55,652 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:13:55,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:13:56,004 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:13:58,006 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:00,009 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:01,148 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:01,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:02,012 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:04,014 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:06,017 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:06,149 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:06,707 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:08,019 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:10,021 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:10,717 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:14:12,024 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:12,233 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:14,026 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:16,028 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:16,150 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:17,768 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:18,030 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:20,033 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:21,151 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:22,035 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:23,303 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:23,783 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:14:24,038 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:25,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:14:26,040 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:28,042 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:28,825 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:30,045 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:31,153 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:32,047 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:34,050 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:34,365 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:36,052 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:36,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:38,054 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:39,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:40,057 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:40,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:14:42,059 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:44,062 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:45,439 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:46,064 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:46,155 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:48,066 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:50,069 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:50,983 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:51,156 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:14:52,071 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:53,785 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:14:54,073 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:55,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:14:56,076 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:14:56,517 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:14:58,078 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:00,080 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:01,158 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:02,051 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:02,083 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:04,085 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:06,088 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:06,159 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:07,596 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:08,091 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:10,093 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:10,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:15:12,095 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:13,114 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:14,098 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:16,100 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:16,161 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:18,103 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:18,660 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:20,105 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:21,162 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:22,107 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:23,790 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:15:23,791 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:24,109 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:25,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:15:26,112 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:28,114 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:29,744 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:30,117 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:31,163 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:32,119 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:34,122 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:35,280 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:36,124 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:36,164 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:38,126 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:40,129 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:40,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:15:40,827 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:42,131 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:44,134 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:46,136 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:46,166 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:46,361 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:48,140 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:50,142 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:51,167 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:15:51,908 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:52,144 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:53,791 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:15:54,146 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:55,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:15:56,149 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:15:57,445 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:15:58,151 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:00,153 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:01,170 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:02,156 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:02,986 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:04,158 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:06,161 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:06,171 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:08,163 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:08,523 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:10,165 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:10,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:16:12,168 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:14,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:14,170 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:16,172 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:16,173 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:18,175 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:19,601 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:20,177 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:21,173 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:22,180 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:23,794 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:16:24,182 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:25,143 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:25,718 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:16:26,185 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:28,187 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:30,189 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:30,690 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:31,175 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:32,191 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:34,194 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:36,176 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:36,196 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:36,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:38,199 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:40,201 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:40,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:16:41,775 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:42,204 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:44,207 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:46,178 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:46,209 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:47,311 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:48,211 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:50,213 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:51,179 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:16:52,216 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:52,851 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:16:53,796 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:16:54,218 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:55,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:16:56,220 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:58,222 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:16:58,385 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:00,225 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:01,180 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:02,227 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:03,917 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:04,229 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:06,181 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:06,232 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:08,234 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:09,454 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:10,237 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:10,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:17:12,239 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:14,242 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:15,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:16,183 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:16,244 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:18,246 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:20,248 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:20,536 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:21,184 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:22,251 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:23,799 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:17:24,253 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:25,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:17:26,076 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:26,255 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:28,258 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:30,260 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:31,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:31,597 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:32,262 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:34,264 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:36,187 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:36,267 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:37,130 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:38,269 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:40,271 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:40,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:17:42,274 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:42,662 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:44,276 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:46,188 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:46,278 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:48,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:48,280 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:50,283 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:51,189 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:17:52,285 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:53,726 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:17:53,802 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:17:54,288 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:55,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:17:56,290 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:58,292 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:17:59,254 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:00,294 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:01,191 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:02,297 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:04,300 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:04,792 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:06,192 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:06,302 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:08,304 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:10,307 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:10,326 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:10,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:18:12,309 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:14,312 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:15,867 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:16,193 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:16,314 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:18,316 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:20,319 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:21,195 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:21,403 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:22,321 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:23,805 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:18:24,323 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:25,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:18:26,326 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:26,927 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:28,328 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:30,330 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:31,196 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:32,333 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:32,465 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:34,335 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:36,198 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:36,338 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:37,996 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:38,340 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:40,343 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:40,719 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:18:42,345 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:43,531 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:44,348 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:46,199 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:46,350 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:48,352 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:49,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:50,355 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:51,200 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:18:52,357 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:53,809 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:18:54,359 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:54,811 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:18:55,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:18:56,362 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:18:58,364 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:00,154 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:00,366 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:01,202 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:02,369 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:04,371 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:05,702 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:06,203 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:06,373 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:08,375 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:10,378 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:10,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:19:11,242 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:12,380 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:14,383 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:16,204 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:16,385 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:16,788 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:18,387 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:20,389 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:21,205 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:22,331 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:22,391 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:23,810 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:19:24,394 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:25,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:19:26,396 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:27,873 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:28,398 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:30,401 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:31,207 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:32,403 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:33,406 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:34,406 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:36,208 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:36,408 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:38,410 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:38,953 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:40,413 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:40,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:19:42,415 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:44,418 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:44,493 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:46,210 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:46,421 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:48,423 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:50,004 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:50,425 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:51,211 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:19:52,428 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:53,813 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:19:54,430 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:55,534 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:19:55,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:19:56,433 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:19:58,435 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:00,437 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:01,063 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:01,212 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:02,440 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:04,442 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:06,213 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:06,444 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:06,587 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:08,447 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:10,449 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:10,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:20:12,115 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:12,452 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:14,454 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:16,215 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:16,456 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:17,650 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:18,459 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:20,461 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:21,216 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:22,464 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:23,186 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:23,848 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:20:24,466 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:25,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:20:26,469 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:28,471 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:28,734 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:30,473 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:31,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:32,476 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:34,272 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:34,478 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:36,218 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:36,480 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:38,482 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:39,802 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:40,485 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:40,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:20:42,487 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:44,489 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:44,958 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:46,220 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:46,492 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:48,494 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:50,257 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:50,497 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:51,221 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:20:52,499 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:53,850 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:20:54,502 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:55,556 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:20:55,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:20:56,504 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:20:58,507 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:00,509 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:00,857 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:01,223 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:02,511 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:04,514 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:06,157 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:06,224 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:06,516 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:08,519 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:10,521 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:10,721 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:21:11,457 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:12,524 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:14,526 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:16,228 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:16,528 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:16,757 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:18,531 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:20,533 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:21,229 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:22,056 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:22,536 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:23,854 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:21:24,538 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:25,720 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:21:26,765 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:21:26,787 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 11:21:26,794 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 11:21:26,803 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 11:21:27,542 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 11:21:27,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:28,543 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:30,989 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:31,414 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:32,809 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:36,615 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:37,810 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:40,557 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:41,397 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:21:42,819 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:46,548 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:47,820 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:52,467 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:52,821 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:52,945 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:21:52,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:21:53,327 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: log_artifact 2025-09-18 11:21:53,327 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: log_artifact 2025-09-18 11:21:53,345 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: partial_history 2025-09-18 11:21:53,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_20_4afbe92e9e9879d42393.table.json 2025-09-18 11:21:53,574 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table 2025-09-18 11:21:54,090 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/i9ijeqp1-media/table/rollouts_20_4afbe92e9e9879d42393.table.json 2025-09-18 11:21:54,388 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: poll_exit 2025-09-18 11:21:54,575 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:54,663 INFO wandb-upload_0:1208566 [upload_job.py:push():89] Uploaded file /global/homes/a/amahesh/.local/share/wandb/artifacts/staging/tmpz2dmxxvt 2025-09-18 11:21:55,953 INFO SenderThread:1208566 [sender.py:send_request_log_artifact():1458] logged artifact run-jdwsvk36-rollouts - {'id': 'QXJ0aWZhY3Q6MjA2NDE0NTY1NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc0NTM0MDUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MjA2Mzg2NTY5OQ==', 'versionIndex': 18}}} 2025-09-18 11:21:55,954 DEBUG SenderThread:1208566 [sender.py:send():382] send: files 2025-09-18 11:21:55,954 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file media/table/rollouts_20_4afbe92e9e9879d42393.table.json with policy now 2025-09-18 11:21:55,954 DEBUG SenderThread:1208566 [sender.py:send():382] send: exit 2025-09-18 11:21:55,955 INFO SenderThread:1208566 [sender.py:send_exit():589] handling exit code: 0 2025-09-18 11:21:55,955 INFO SenderThread:1208566 [sender.py:send_exit():591] handling runtime: 48214 2025-09-18 11:21:55,964 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 11:21:55,964 INFO SenderThread:1208566 [sender.py:send_exit():597] send defer 2025-09-18 11:21:55,964 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:21:55,964 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:55,965 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: poll_exit 2025-09-18 11:21:55,965 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 0 2025-09-18 11:21:55,966 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:55,966 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 0 2025-09-18 11:21:55,966 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 1 2025-09-18 11:21:55,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:55,966 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 1 2025-09-18 11:21:55,966 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:55,966 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 1 2025-09-18 11:21:55,966 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 2 2025-09-18 11:21:55,966 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:55,966 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 2 2025-09-18 11:21:55,966 INFO HandlerThread:1208566 [system_monitor.py:finish():203] Stopping system monitor 2025-09-18 11:21:55,967 DEBUG SystemMonitor:1208566 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-09-18 11:21:55,967 INFO HandlerThread:1208566 [interfaces.py:finish():202] Joined cpu monitor 2025-09-18 11:21:55,967 DEBUG SystemMonitor:1208566 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-09-18 11:21:55,967 INFO HandlerThread:1208566 [interfaces.py:finish():202] Joined disk monitor 2025-09-18 11:21:56,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 11:21:56,578 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:56,668 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /tmp/tmp811hsqpbwandb/v7iuvk3m-media/table/rollouts_20_4afbe92e9e9879d42393.table.json 2025-09-18 11:21:57,377 INFO HandlerThread:1208566 [interfaces.py:finish():202] Joined gpu monitor 2025-09-18 11:21:57,377 INFO HandlerThread:1208566 [interfaces.py:finish():202] Joined memory monitor 2025-09-18 11:21:57,377 INFO HandlerThread:1208566 [interfaces.py:finish():202] Joined network monitor 2025-09-18 11:21:57,379 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:57,379 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 2 2025-09-18 11:21:57,379 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 3 2025-09-18 11:21:57,379 DEBUG SenderThread:1208566 [sender.py:send():382] send: stats 2025-09-18 11:21:57,379 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:57,380 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 3 2025-09-18 11:21:57,381 DEBUG SenderThread:1208566 [sender.py:send():382] send: history 2025-09-18 11:21:57,381 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: summary_record 2025-09-18 11:21:57,389 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 11:21:57,390 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:57,390 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 3 2025-09-18 11:21:57,390 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 4 2025-09-18 11:21:57,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:57,390 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 4 2025-09-18 11:21:57,390 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:57,390 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 4 2025-09-18 11:21:57,390 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 5 2025-09-18 11:21:57,390 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:57,391 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 5 2025-09-18 11:21:57,401 DEBUG SenderThread:1208566 [sender.py:send():382] send: summary 2025-09-18 11:21:57,411 INFO SenderThread:1208566 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2025-09-18 11:21:57,411 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:57,412 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 5 2025-09-18 11:21:57,412 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 6 2025-09-18 11:21:57,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:57,412 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 6 2025-09-18 11:21:57,412 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:57,412 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 6 2025-09-18 11:21:57,412 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 7 2025-09-18 11:21:57,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: status_report 2025-09-18 11:21:57,412 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:57,413 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 7 2025-09-18 11:21:57,413 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:57,413 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 7 2025-09-18 11:21:57,580 INFO Thread-12 :1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 11:21:58,505 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 8 2025-09-18 11:21:58,505 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:58,505 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 8 2025-09-18 11:21:58,505 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:58,505 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 8 2025-09-18 11:21:58,505 INFO SenderThread:1208566 [job_builder.py:build():298] Attempting to build job artifact 2025-09-18 11:21:58,529 INFO SenderThread:1208566 [job_builder.py:_get_source_type():428] is repo sourced job 2025-09-18 11:21:58,529 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 9 2025-09-18 11:21:58,529 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:58,529 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 9 2025-09-18 11:21:58,530 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:58,530 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 9 2025-09-18 11:21:58,530 INFO SenderThread:1208566 [dir_watcher.py:finish():358] shutting down directory watcher 2025-09-18 11:21:58,582 INFO SenderThread:1208566 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:58,582 INFO SenderThread:1208566 [dir_watcher.py:finish():388] scan: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files 2025-09-18 11:21:58,582 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-metadata.json wandb-metadata.json 2025-09-18 11:21:58,582 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/requirements.txt requirements.txt 2025-09-18 11:21:58,582 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log output.log 2025-09-18 11:21:58,582 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/config.yaml config.yaml 2025-09-18 11:21:58,586 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json wandb-summary.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/videos/Inference samples_0_07b9411d7e9b5ed6f3a1.gif media/videos/Inference samples_0_07b9411d7e9b5ed6f3a1.gif 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_1_83148a7870823ece0187.table.json media/table/rollouts_1_83148a7870823ece0187.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_11_aa86ab67448a943ea3cd.table.json media/table/rollouts_11_aa86ab67448a943ea3cd.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_10_1700d5d5174101ae4c4f.table.json media/table/rollouts_10_1700d5d5174101ae4c4f.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_20_4afbe92e9e9879d42393.table.json media/table/rollouts_20_4afbe92e9e9879d42393.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_9_570b2d39ebd6823f0cfe.table.json media/table/rollouts_9_570b2d39ebd6823f0cfe.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_14_219cd90375b01ac0f3dc.table.json media/table/rollouts_14_219cd90375b01ac0f3dc.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_12_00b26c37b2bb10ac0977.table.json media/table/rollouts_12_00b26c37b2bb10ac0977.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_8_9b215a1de50b7a925101.table.json media/table/rollouts_8_9b215a1de50b7a925101.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_17_97878f334b2dd769a034.table.json media/table/rollouts_17_97878f334b2dd769a034.table.json 2025-09-18 11:21:58,590 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_16_84d500de1eb4d485000c.table.json media/table/rollouts_16_84d500de1eb4d485000c.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_3_71cc41780ac1c30b1113.table.json media/table/rollouts_3_71cc41780ac1c30b1113.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_19_ee0d3182ce72199f9003.table.json media/table/rollouts_19_ee0d3182ce72199f9003.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_18_b125d777d8ca0453b1d6.table.json media/table/rollouts_18_b125d777d8ca0453b1d6.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_2_239e091867f620d0e20a.table.json media/table/rollouts_2_239e091867f620d0e20a.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_13_1528d88fda0ca905ee0d.table.json media/table/rollouts_13_1528d88fda0ca905ee0d.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_6_6e93dc5c1a3f79a3476b.table.json media/table/rollouts_6_6e93dc5c1a3f79a3476b.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_4_1db01a832c96ad087afc.table.json media/table/rollouts_4_1db01a832c96ad087afc.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_5_1a5c30fa6dafb36e5691.table.json media/table/rollouts_5_1a5c30fa6dafb36e5691.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_7_15593048a2d6b7646c62.table.json media/table/rollouts_7_15593048a2d6b7646c62.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [dir_watcher.py:finish():402] scan save: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/media/table/rollouts_15_1d2bb76f3e4665f270d8.table.json media/table/rollouts_15_1d2bb76f3e4665f270d8.table.json 2025-09-18 11:21:58,591 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 10 2025-09-18 11:21:58,592 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:58,592 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 10 2025-09-18 11:21:58,592 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:58,592 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 10 2025-09-18 11:21:58,592 INFO SenderThread:1208566 [file_pusher.py:finish():175] shutting down file pusher 2025-09-18 11:21:58,887 INFO wandb-upload_1:1208566 [upload_job.py:push():131] Uploaded file /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/output.log 2025-09-18 11:21:59,084 INFO wandb-upload_0:1208566 [upload_job.py:push():131] Uploaded file /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/requirements.txt 2025-09-18 11:21:59,182 INFO wandb-upload_3:1208566 [upload_job.py:push():131] Uploaded file /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/wandb-summary.json 2025-09-18 11:21:59,219 INFO wandb-upload_2:1208566 [upload_job.py:push():131] Uploaded file /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/files/config.yaml 2025-09-18 11:21:59,389 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: keepalive 2025-09-18 11:21:59,420 INFO Thread-11 (_thread_body):1208566 [sender.py:transition_state():617] send defer: 11 2025-09-18 11:21:59,420 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:59,421 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 11 2025-09-18 11:21:59,421 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:59,421 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 11 2025-09-18 11:21:59,421 INFO SenderThread:1208566 [file_pusher.py:join():181] waiting for file pusher 2025-09-18 11:21:59,421 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 12 2025-09-18 11:21:59,421 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:59,421 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 12 2025-09-18 11:21:59,421 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:59,422 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 12 2025-09-18 11:21:59,422 INFO SenderThread:1208566 [file_stream.py:finish():595] file stream finish called 2025-09-18 11:21:59,937 INFO SenderThread:1208566 [file_stream.py:finish():599] file stream finish is done 2025-09-18 11:21:59,937 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 13 2025-09-18 11:21:59,937 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:59,937 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 13 2025-09-18 11:21:59,937 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:59,938 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 13 2025-09-18 11:21:59,938 INFO SenderThread:1208566 [sender.py:transition_state():617] send defer: 14 2025-09-18 11:21:59,938 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: defer 2025-09-18 11:21:59,938 DEBUG SenderThread:1208566 [sender.py:send():382] send: final 2025-09-18 11:21:59,938 INFO HandlerThread:1208566 [handler.py:handle_request_defer():172] handle defer: 14 2025-09-18 11:21:59,939 DEBUG SenderThread:1208566 [sender.py:send():382] send: footer 2025-09-18 11:21:59,939 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: defer 2025-09-18 11:21:59,939 INFO SenderThread:1208566 [sender.py:send_request_defer():613] handle sender defer: 14 2025-09-18 11:21:59,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: poll_exit 2025-09-18 11:21:59,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: server_info 2025-09-18 11:21:59,940 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: get_summary 2025-09-18 11:21:59,940 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: poll_exit 2025-09-18 11:21:59,946 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: sampled_history 2025-09-18 11:21:59,946 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: server_info 2025-09-18 11:21:59,947 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-18 11:21:59,949 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: job_info 2025-09-18 11:22:00,048 DEBUG SenderThread:1208566 [sender.py:send_request():409] send_request: job_info 2025-09-18 11:22:00,049 INFO MainThread:1208566 [wandb_run.py:_footer_history_summary_info():3837] rendering history 2025-09-18 11:22:00,052 INFO MainThread:1208566 [wandb_run.py:_footer_history_summary_info():3869] rendering summary 2025-09-18 11:22:00,053 INFO MainThread:1208566 [wandb_run.py:_footer_sync_info():3796] logging synced files 2025-09-18 11:22:00,053 DEBUG HandlerThread:1208566 [handler.py:handle_request():146] handle_request: shutdown 2025-09-18 11:22:00,053 INFO HandlerThread:1208566 [handler.py:finish():866] shutting down handler 2025-09-18 11:22:00,950 INFO WriterThread:1208566 [datastore.py:close():294] close: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/run-jdwsvk36.wandb 2025-09-18 11:22:01,049 INFO SenderThread:1208566 [sender.py:finish():1548] shutting down sender 2025-09-18 11:22:01,049 INFO SenderThread:1208566 [file_pusher.py:finish():175] shutting down file pusher 2025-09-18 11:22:01,049 INFO SenderThread:1208566 [file_pusher.py:join():181] waiting for file pusher